Skip to content

Commit

Permalink
Merge pull request #970 from lk-iqt/doc-updates
Browse files Browse the repository at this point in the history 
Documentation updates
  • Loading branch information
@anarkiwi
anarkiwi authored Nov 14, 2023
2 parents d92fc1b + 64461f9 commit 602ea3c
Show file tree
Hide file tree
Showing 10 changed files with 387 additions and 175 deletions.
187 changes: 17 additions & 170 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,186 +23,33 @@ If workers have been provisioned, the sigfinder will then command the workers to

As there will almost certainly be more signals than workers available, sigfinder will prioritize signals that it least often observed over a configurable number of scanner cycles. It is possible to configure this to `off` so that the recording choice will be random. It is also possible to configure the workers to tell the sigfinder to exclude that worker from certain frequency ranges (if for example the worker SDR cannot handle some part of the frequency spectrum scanned).

## Operating gamutRF
## Build and Operating gamutRF

See the [build doc](docs/BUILD.md)
See the [build doc](docs/2-BUILD.md) for instruction on setting up a GamutRF system. Next see the [operating instructions](docs/4-OPERATION.md) for instructions on operating GamutRF.

### SDR/scanner/sigfinder command line options
## Contributing

While there are other options, these options primarily influence gamutRF's scanner functionality.

#### scanner

| Option | Description |
| -- | -- |
| --freq-start and --freq-end | Start and end of frequency range to scan in Hz |
| --tuning_ranges | Overrides --freq-start and --freq-end if present. A comma separated list of ranges in Hz to scan, for example ```2.2e9-2.6e9,5.1e9-5.9e9``` |
| --igain | SDR input gain in dB |
| --samp-rate | Number of samples/sec |
| --tuneoverlap | For each retune interval, advance center frequency by N * --samp_rate |
| --sweep-sec | Attempt to sweep frequency range within N seconds (effectively, best guess for --tune-step-fft) |
| --tune-step-fft | Overrides --sweep-sec if present. Retune to next interval every N FFT points |
| --skip-tune-step | Discard N FFT points after retuning |
| --nfft | Number of FFT points |
| --write_samples | If > 0, write N samples to disk after retuning, in --sample_dir as a zstd compressed file |
| --description | Optionally provide text description along with streaming scanner updates, to the sigfinder |

##### sigfinder

| Option | Description |
| -- | -- |
| --bin_width | Bin width in MHz |
| --history | Number of scanner cycles over which to prioritize recording of least often observed peaks |
| --record_bw_msps | Number of samples per second in units of 1024^2 (generally larger than bin size to record signal margins) |
| --record_secs | Length of time to record I/Q samples in seconds |
| --fftlog | Log raw output of CSV to this file, which will be rotated every --rotatesecs |
| --scanners | Comma separated list of scanner ZMQ endpoints to use, for example --scanners=127.0.0.1:8001,127.0.0.1:9002 |

### Running multiple radios on the same machine

gamutRF supports the use of multiple radios on the same machine, whether for scanning or recording. When using multiple Ettus SDRs, assign a specific radio to a specific container by specifying the ```serial``` UHD driver argument. To list all connected Ettus radios, run ```uhd_find_devices```. Then add the ```--sdrargs``` argument to the specific container. For example, ```--sdrargs num_recv_frames=960,recv_frame_size=16360,type=b200,serial=12345678```.

gamutRF also supports the KrakenSDR (which presents as five librtlsdr radios). You can run a scanner container for each radio, by adding a serial number - for example, ```--sdr=rtlsdr,serial=1000```. Use ```SoapySDRUtil --find``` to check the radios are correctly connected.

### Manually initiating worker actions

The orchestrator has a web interface on port 80. You can use this to command a worker to start an I/Q sample recording or start a RSSI stream.

You can also call the worker API directly (for example, to use a stand-alone worker to make a recording or stream RSSI).

For example, make a recording with a sample rate of 20.48e6, for 2 seconds (409600000 samples worth), at 100MHz:

```
$ wget -nv -O- localhost:8000/v1/record/100000000/409600000/20480000
```

To stream RSSI values instead, call:

```
$ wget -nv -O- localhost:8000/v1/rssi/100000000/409600000/20480000
```

If the sample count parameter is 0, the stream will not end
until a new RPC (whether rssi or record) is received.

## Working with worker I/Q recordings

Workers make recordings that are compressed with zstandard, and are typically in complex number, int16 format, and include the center frequency and sample rate that the recording was made with. gamutRF tools can generally work with such files directly, but other tools require the recordings to be converted (see below).

### Generating a spectrogram of a recording

gamutRF provides a tool to convert a recording or directory of recordings into a spectrogram. For example, to convert all I/Q recordings in /tmp:

```docker run -ti -v /tmp:/tmp iqtlabs/gamutrf gamutrf-specgram /tmp```

Use the ```--help``` option to change how the spectogram is generated (for example, to change the sample rate).

### Translating recordings to "gnuradio" format

Most SDR tools by convention take an uncompressed raw binary file as input, of [gnuradio type complex](https://blog.sdr.hu/grblocks/types.html). The user must explicitly specify to most SDR tools what sample rate the file was made at to correctly process it. gamutRF provides a tool that converts a gamutRF I/Q recording (which may be compressed) to an uncompressed binary file. For example:

```
docker run -v /tmp:/tmp -ti iqtlabs/gamutrf gamutrf-samples2raw /tmp/gamutrf_recording_ettus_directional_gain70_1234_100000000Hz_20971520sps.s16.zst
```

### Reviewing a recording interactively in gqrx

[gqrx](https://gqrx.dk/) is a multiplatform open source tool that allows some basic SDR operations like visualizing or audio demodulating an I/Q sample recording (see the [github releases page](https://github.com/gqrx-sdr/gqrx/releases), for a MacOS .dmg file). To use gqrx with a gamutRF recording, first translate the recording to gnuradio format (see above). Then open gqrx.

* Select ```Complex Sampled (I/Q) File```
* Set ```Input rate``` to be the same as the gamutRF sample rate (e.g. from the recording file name,
```gamutrf_recording_ettus_directional_gain70_1234_100000000Hz_20971520sps.raw```, set ```Input rate``` to 20971520, and also edit ```rate=``` in ```Device string``` to be 20971520)
* Set ``Bandwidth`` to 0
* Edit ```Device string``` to set the ```file=``` to be the path to the recording.
* Set ```Decimation``` to None.
* Finally select ```OK``` and then ```play``` from the gqrx interface to watch the recording play.

### Reducing recording sample rate

You may want to reduce the sample rate of a recording or re-center it with respect to frequency (e.g. to use another demodulator tool that doesn't support a high sample rate). gamutRF provides the ```freqxlator``` tool to do this.

* Translate your gamutRF recording to gnuradio format (see above).
* Use ```freqxlator``` to create a new recording at a lower sample rate, potentially with a different center frequency.

For example, to reduce a recording made with gamutRF's default sample rate to 1/10th the rate while adjusting the center frequency down by 1MHz, use:

```docker run -ti iqtlabs/gamutrf gamutrf-freqxlator --samp-rate 20971520 --center -1e6 --dec 10 gamutrf_recording_gain70_1234_100000000Hz_20971520sps.raw gamutrf_recording_gain70_1234_100000000Hz_2097152sps.raw```

### Demodulating AM/FM audio from a recording

gamutRF provides a tool to demodulate AM/FM audio from a recording as an example use case.

* Use the ```freqxlator``` tool to make a new recording at no more than 1Msps and has the frequency to be demodulated centered.
* Use the ```airspyfm``` tool to demodulate audio to a WAV file.

For example, to decode an FM recording which must be at the center frequency of a recording:

```docker run -v /tmp:/tmp -ti iqtlabs/gamutrf-airspyfm -m fm -t filesource -c filename=/tmp/gamutrf_recording_gain70_1234_100000000Hz_2097152sps.raw,raw,format=FLOAT,srate=2097152 -F /tmp/out.wav```

Run:

```docker run -ti iqtlabs/gamutrf-airspyfm -h```

To view other options.

## API access

gamutRF supports two separate APIs - for receiving scanner updates, and making scanner configuration changes.

* scanner update API: described in [zmqreceiver.py](gamutrf/zmqreceiver.py). Receives real time scanner updates and config.
* scanner config API: allows RESTful updates to any CLI argument. Eg, ```wget -O- "http://localhost:9001/reconf?freq_start=1e9&freq_end=2e9"``` causes the scanner to reset and scan 1e9 to 2e9Hz. Any config change causes the scanner's gnuradio flowgraph to restart.

## Scanner testing

Currently, the scanner ```gain``` must be set manually for the general RF environment (e.g. noisy/many signals versus quiet/few signals).
To establish the correct values and to confirm the scanner is working, initiate a scan over the 2.3-2.6GHz range. As the 2.4GHz spectrum is very busy with legacy WiFi
and BlueTooth, the probability of seeing signals is high. If in an environment without BlueTooth or WiFi, an alternative is the FM broadcast band (88MHz to 108MHz).

To begin, commence scanning with just the scanner and waterfall containers:
1. Fork the Project
2. Create your Feature Branch (`git checkout -b dev`)
3. Commit your Changes (`git commit -m 'adding some feature'`)
4. Run (and make sure they pass):

```
VOL_PREFIX=/tmp FREQ_START=2.3e9 FREQ_END=2.6e9 docker compose -f orchestrator.yml up gamutrf waterfall
black --diff --check gamutrf
```

Browse the waterfall container's webserver port (by default, localhost:9003). It should contain strong signals similar to this example:

![2.4G example](docs/fft24test.png)

If no or only small peaks appear which are not marked as peaks, increase ```gain``` (e.g., from 40 to 45) until peaks are detected.

If no peaks appear still, check antenna cabling, or choose a different scan range where signals are expected in your environment.

## Troubleshooting

#### SoapySDR errors allocating buffers

Run ```echo 0 > /sys/module/usbcore/parameters/usbfs_memory_mb``` as root before starting the scanner(s).
5. Push to the Branch (`git push origin dev`)
6. Open a Pull Request

#### Containers won't start using Ettus SDRs
See `CONTRIBUTING.md` for more information.

##### ```[ERROR] [USB] USB open failed: insufficient permissions```

Ettus SDRs download firmware and switch USB identities when first powered up. Restart the affected container to work around this (if run with docker compose, restart will happen automatically).

##### ```[ERROR] [UHD] An unexpected exception was caught in a task loop.The task loop will now exit, things may not work.boost: mutex lock failed in pthread_mutex_lock: Invalid argument```

UHD driver arguments ```num_recv_frames``` or ```recv_frame_size``` may be too high. The defaults are defined as ETTUS_ARGS in [utils.py](gamutrf/utils.py). Try reducing one or both via ```--sdrargs```. For example, ```--sdrargs num_recv_frames=64,recv_frame_size=8200,type=b200```.

#### ```[ERROR] [UHD] EnvironmentError: IOError: usb rx6 transfer status: LIBUSB_TRANSFER_OVERFLOW```

Stop containers, and reset the Ettus as follows:

```
$ /usr/lib/uhd/utils/b2xx_fx3_utils -D
$ /usr/lib/uhd/utils/b2xx_fx3_utils -U
$ /usr/lib/uhd/utils/b2xx_fx3_utils -S
```
## License

#### Scanner with Ettus SDR shows implausible low power at approx 100MHz intervals
Distributed under the [Apache 2.0](https://github.com/IQTLabs/aisonobuoy-data-science/blob/main/LICENSE.txt). See [LICENSE](./LICENSE) for more information.

Ettus radios periodically need extra time to produce good data when being retuned rapidly by the scanner. Increasing the value of ```--db_clamp_floor``` will cause the scanner to discard windows after retuning (effectively waiting for the retune command to be executed and produce good data before proceeding).
## Contact IQTLabs

#### "O"s or warnings about overflows in SDR containers
- Twitter: [@iqtlabs](https://twitter.com/iqtlabs)
- Email: [email protected]

* Ensure your hardware can support the I/Q sample rate you have configured (gamutRF has been tested on Pi4 at 20Msps, which is the default recording rate). Also ensure your recording medium (e.g. flash drive, USB hard disk) is not timing out or blocking.
* If using a Pi4, make sure you are using active cooling and an adequate power supply (no CPU throttling), and you are using a "blue" USB3 port.
See our other projects: [https://github.com/IQTLabs/](https://github.com/IQTLabs/)
7 changes: 7 additions & 0 deletions SECURITY.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Security Policy

IQT Labs takes the security of its projects very seriously. If you find a vulnerability or a bug with security implications we want to hear about it. That being said, please **do not** submit an issue via the standard issue tracker as these are publicly visible. Instead we ask that you email your findings to our security team at [[email protected]](mailto:[email protected]) so that we can properly assess and address these issues. Please err on the side of caution, even partial or fragmentary information may be helpful to us.

We will attempt to confirm all reports within 72 hours. Triage and remediation timelines will depend on the nature and complexity of the issue found.

We appreciate the efforts of anyone attempting to help improve the security of our tools!
7 changes: 7 additions & 0 deletions docs/1-INTRODUCTION.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Introduction to GamutRF

GamutRF is a combination of open-source software based on the GNU radio framework and low-cost software defined radios, that together implements radio spectrum scanning, finding, and recording raw signals, with signal identification and geolocalization using a variety of machine learning approaches. GamutRF can be run on a set of Raspberry Pi4s working together in a mobile/battery powered system, can scale up to more powerful compute platforms, can offload processing to GPUs, and can support multiple radios and distributed signal finding and recording.

At its core, the project harnesses SDR platforms that are dynamically retuned across a predefined bandwidth and collating the disparate signals into one cohesive picture, thereby achieving ultrawide spectrum awareness. By giving an ultra-wide stare at the entire usable spectrum, we are better able to understand spectrum utilization, and both common and uncommon RF transmissions. This combination of classification and localization provides numerous opportunities for better understanding the RF spectrum in both time and space.

The signals of interest are discerned through a two-pronged approach: energy spectra analysis and RF spectrogram-based machine learning classification. The energy spectra analysis involves identifying unusual spikes or patterns in signal energy across the spectrum. This aids in quickly flagging known signals that warrant further investigation. Concurrently, the RF spectrogram-based machine learning classification brings a new dimension to signal identification. By generating and analyzing spectrograms, which represent the spectral content of a signal over time, the system employs machine learning models to classify and categorize signals. This fusion of techniques not only expedites the identification process but also ensures a comprehensive understanding of the RF environment.
Loading

0 comments on commit 602ea3c

Please sign in to comment.