If already have a version of the Sen2-Agri system installed, you can easily upgrade it to a newer version by downloading the new installation package and running the update script. To do so, you should navigate to the installation package directory and start the process:
cd ~/Downloads/Sen2AgriDistribution/install_script
sudo ./update.sh
See the MACCS update section for instructions about updating MACCS.
Starting from version 1.6, MACCS is no longer provided with the Sen2-Agri installation files. You will therefore have to download it from the CNES website and install it prior the Sen2-Agri installation in itself. Steps are also detailed in the Software User Manual (p. 30). Basically, you should:
- Go to https://logiciels.cnes.fr/content/maccs?language=en.
- Click on the "Télécharger" or "Download" button in the tab with the same name
- Read the license agreement carefully
- Fill the form with your information using "Sentinel-2 for Agriculture" as the project name
- Click on the "I accept the license" checkbox and then on the button that appears. The download should now start.
Once you have downloaded the MACCS installation package, you can follow the installation process detailed in its manual. You can find the MACCS manual in the installation package that you just downloaded. The installation process is summarized here:
# extract the package
unzip MACCS-site-CNES.zip
tar xf maccs-5.1.5-centos.7.2.1511.x86_64-release-gcc.TGZ.tar
tar xf maccs-cots-4.3.1-rhel.7.2.x86_64-release-gcc.tar
# install its dependencies (COTS)
cd maccs-cots-4.3.1-rhel.7.2.x86_64-release-gcc
sudo ./install-maccs-cots.sh
# install MACCS
cd ..
sudo ./install-maccs-5.1.5-centos.7.2.1511.x86_64-release-gcc.TGZ.sh
# if you your system was previously working with a older MACCS version, you will also have to update the MACCS location in the Sen2-Agri database
sudo -u postgres psql sen2agri -c \
"update config set value = '/opt/maccs/core/5.1/bin/maccs' \
where key = 'demmaccs.maccs-launcher';"
Starting from version 1.8 a new service was introduced in the Sen2Agri system. The role of the services is: * Replace the downloaders implementation. This means that the sen2agri-{sentinel,landsat}-downloader.{service,timer} services are no longer available in the version 1.8. * During download, a datasource can be used for querying the list of products and another datasource can be used for actually downloading the products. For example, for S2 the SciHub can be used for querying the list of products and Amazon or a local store like the IPT can be used for downloading or just creating symlinks to the products (for local stores). * Provide an API for the GUI for various operations (stop/start downloads, delete site, monitoring etc.)
The configuration of the sen2agri-services can be done via:
- The /usr/share/sen2agri/sen2agri-services/config/sen2agri-services.properties
- Advanced configuration via the "datasource" table in the database.
In some specific scenarios (e.g. when updating MACCS) you might want to temporarily stop the Sen2-Agri services. You can do this with the following command:
sudo systemctl stop \
sen2agri-monitor-agent sen2agri-scheduler sen2agri-executor \
sen2agri-orchestrator sen2agri-http-listener \
sen2agri-{sentinel,landsat}-downloader.{service,timer} \
sen2agri-demmaccs.{service,timer} sen2agri-services
To restart them you can use a similar command:
sudo systemctl start \
sen2agri-monitor-agent sen2agri-scheduler sen2agri-executor \
sen2agri-orchestrator sen2agri-http-listener \
sen2agri-{sentinel,landsat}-downloader.{service,timer} \
sen2agri-demmaccs.{service,timer} sen2agri-services
There is no need to.
-
Optionally, remove the
PostgreSQL
databasesudo -u postgres psql -c "drop database sen2agri;"
-
Stop
SLURM
,MariaDB
andPostgreSQL
sudo systemctl stop slurmd slurmctld slurmdbd munge mariadb postgresql-9.4 sudo systemctl disable slurmd slurmctld slurmdbd munge mariadb postgresql-9.4
-
Uninstall the Sen2-Agri packages
sudo yum remove \ sen2agri-processors sen2agri-website sen2agri-downloaders-demmaccs \ sen2agri-app
-
Uninstall
SLURM
,MariaDB
andPostgreSQL
sudo yum remove \ slurm slurm-munge slurm-slurmdb-direct slurm-sql slurm-torque slurm-sjstat \ slurm-plugins slurm-slurmdbd slurm-devel slurm-perlapi slurm-sjobexit \ slurm-pam_slurm munge munge-devel munge-libs mariadb-server mariadb-devel \ mariadb postgresql94-server postgresql94-contrib postgresql94
-
Stop and disable the web server
sudo systemctl stop http sudo systemctl disable http
-
Uninstall the web server and
PHP
sudo yum remove httpd php
-
Uninstall the sen2agri-services
sudo systemctl stop sen2agri-services sudo rm -fR /usr/share/sen2agri/sen2agri-services
-
Uninstall
OTB
sudo yum remove otb
-
Uninstall the other dependencies
sudo yum remove \ boost tinyxml tinyxml-devel qt qt5-qtbase qt5-qtbase-postgresql qt-x11 fftw \ gdal geos libgeotiff libsvm muParser opencv openjpeg2 openjpeg2-tools proj \ proj-epsg swig gsl
Note that other applications on the system might depend on these packages. Before continuing, please make sure to double-check the list of packages to be removed.
-
Uninstall MACCS
sudo rm -rf /opt/maccs
For more information, please consult the MACCS Software User Manual.
-
Optionally, remove the products
sudo rm -rf /mnt/{archive,scratch,upload}
If you're planning to make the web interface available from outside, it's a good idea to use a reverse proxy like nginx
instead of directly exposing the server running the Sen2-Agri system. For example, as mentioned in another section.
For nginx
, the following (untested) configuration snippet can be used as a starting point:
upstream sen2agri {
server 127.0.0.1:8001 fail_timeout=0;
}
server {
listen 80;
server_name sen2agri.host.domain;
root /dev/null;
location / {
proxy_read_timeout 300;
proxy_connect_timeout 300;
proxy_redirect off;
proxy_set_header Host $host;
proxy_pass http://sen2agri;
}
}
It expects the site to be accessible from a subdomain, which simplifies the configuration by avoiding the use of rewrite rules. At this point, the web interface currently does not support running in a subdirectory (it expects to be found under http://sen2agri.example.com/
instead of e.g. http://example.com/sen2agri
).
Depending on your nginx
configuration structure, you can either place that in /etc/nginx/nginx.conf
, or in a new file under /etc/nginx/sites-available
. In the latter case, you should make a symlink to it under /etc/nginx/sites-enabled
.
You will also need to edit /etc/httpd/conf/httpd.conf
and change the port httpd
is listening on (look for the Listen
directive). The snippet above uses 8001
.
After modifying the nginx
settings, you can test them and reload the configuration:
sudo nginx -t
sudo systemctl reload nginx
sudo systemctl reload httpd
For more information about setting up nginx
or another reverse proxy, consult its official documentation or your system administrator.
To check that the system is running, you can use the following commands:
# check the downloader status in versions prior to version 1.8
systemctl status sen2agri-{sentinel,landsat}-downloader.{timer,service}
# check the downloader status in versions starting with version 1.8
systemctl status sen2agri-services
# check the L2A processor status
systemctl status sen2agri-demmaccs.{timer,service}
# check the other services
systemctl status \
sen2agri-executor sen2agri-http-listener sen2agri-monitor-agent \
sen2agri-scheduler sen2agri-orchestrator
For each of the components above you can use journalctl
to see the last log messages, e.g.:
# check today's log messages for the Sentinel-2 downloader in versions prior to version 1.8
journalctl -u sen2agri-sentinel-downloader --since today
# check today's log messages for the Sentinel-2 downloader in versions starting with version 1.8
journalctl -u sen2agri-services --since today
# check the last log messages
journalctl -e
You can also run systemctl
without parameters to quickly check whether any service has crashed.
See also the section on how to check whether the web site is running.
You can use the following commands:
# check the web server status
# if running, it will show up as "active (running)"
systemctl status httpd
# check that the web server is listening on the HTTP port
sudo ss -pantl | grep 80
# check that the sen2agri-services are listening on the 8081 port
sudo ss -pantl | grep 8081
# open the site, either by navigating to it in a browser
# or from the command line, like below:
xdg-open http://localhost/
The default username and password are sen2agri
and sen2agri
. Currently, there is no way to change them from the web interface.
To add users or change a password, you can use the following commands:
# add an user
# role_id is 1 for administrators and 2 for other users
# site_id is a site id to limit access, or NULL
sudo -u postgres psql sen2agri -c \
"select sp_adduser('login', 'email', 'password', role_id :: smallint, site_id);"
# change the password
# first find out the user id
sudo -u postgres psql sen2agri -c 'select id, login from "user";'
sudo -u postgres psql sen2agri -c \
"select \"sp_changepassword\"(
user_id :: smallint,
'old password',
'new password');"
The database connection settings can be found in /etc/sen2agri/sen2agri.conf
.
If you want to control a running job, you first need to find out its identifier. For that, you can use the "Monitoring" tab of the web interface. In this context, a job is only created shortly before it starts running. See the corresponding section for cancelling a scheduled task.
Once you have the id
, you can use the job_operations.py
script to cancel it:
# display the usage information
python job_operations.py -h
python job_operations.py -j JOB_ID -o cancel
python job_operations.py -j JOB_ID -o delete
The difference between the two operations is that cancel
is supposed to stop the job execution, while delete
will remove its tracking information from the system.
You might also want to pause a running job. In this case, use the pause
operation. To start the job again, use the resume
operation.
python job_operations.py -j JOB_ID -o pause
python job_operations.py -j JOB_ID -o resume
Note that the job cancellation, pause and resume functionality is experimental and might not work properly.
Starting from version 1.8 these operations are also available from the Sen2Agri GUI, from the System Overview tab.
Prior to version 1.8, this operation was not available in the web interface. You will need to remove the scheduled task directly from the database, like below:
# first find out the scheduled task id
sudo -u postgres psql sen2agri -c \
"select scheduled_task.id, site.name, scheduled_task.name \
from scheduled_task \
inner join site \
on site.id = site_id;"
# then delete it, don't forget to replace [ID] below
sudo -u postgres psql sen2agri -c \
"delete \
from scheduled_task \
where id = [ID]"
Starting with the version 1.8, this operation can be done from the Sen2Agri GUI, in the Dashboard tab.
If you have L2A or higher-level products from a different Sen2-Agri installation, you can import them into the database using the insert_l2a_product_to_db.py
script. You will be able to associate them to an existing site. Note that the script name is misleading, as it should also work for products of a different level.
You can either import a single product or multiple ones. To import every product in a directory, you should add -m True
in the command line. Otherwise, the path must point to a single product, as the default is -m False
.
There is no filtering functionality available. For that you will need to either move them to another location, or run the script in the single product mode on each of them. An example is provided below.
# display the usage information
insert_l2a_product_to_db.py -h
# import one L2A product to the Sudan site
# use the short name from the web interface
insert_l2a_product_to_db.py -s sudan -p l2a -t l2a -d product_path
# import multiple L2As to the France site
insert_l2a_product_to_db.py -m True -s france -p l2a -t l2a \
-d /mnt/old_products/france/l2a
# import only the Landsat 8 products; their names start with LC8_
for p in /path/to/l2a/LC8_*; do
insert_l2a_product_to_db.py -s mali -p l2a -t l2a -d ${p}
done
Note that this method doesn't insert the products to the downloader_history
table. This means that, if the downloaders are enabled, the existing products might be downloaded again. The following SQL statement copies entries from the product
table to downloader_history
. It might be useful in these situations.
truncate table downloader_history;
insert
into downloader_history(
site_id,
satellite_id,
product_name ,
full_path,
created_timestamp,
status_id,
no_of_retries,
product_date,
orbit_id
)
select site_id,
satellite_id,
replace(replace(name, 'MSIL2A', 'MSIL1C'), '_L2A', ''),
full_path,
inserted_timestamp,
5,
1,
created_timestamp,
orbit_id
from product
where product_type_id = 1;
You should stop the downloader before running it; otherwise it might re-download the products.
Note, however, that this only works for old-style L8
products. For the new naming convention (e.g. LC08_L1TP_171058_20161011_20170320_01_T1
), the L2A
string needs to be replaced with either L1TP
, L1GS
or L1GT
. Since there is no way to undo the transformation, the only approach that works would be to insert all three.
Prior to versio 1.8, the web interface didn't allow the user to delete a site. If you want to remove a site, you can use the delete_site.py
script.
# display the usage information
delete_site.py -h
# delete the "Belgium" site, but keep its products
# use the short name from the web interface
delete_site.py -s belgium
# delete the "Test Site" site, together with the corresponding products
delete_site.py -s test_site -d false -e false -a false -l false -m false -t false
You can find the short name of the site in the web interface or by looking at the path of its products. Generally, the short name is generated by converting the site name to lower case and replacing spaces with underscores.
Starting with version 1.8, a site can be deleted from its edit page.
To change a site's footprint, you can update the geog
column of the site's site
table entry. The footprint is stored as an EPSG:4326
MULTIPOLYGON
feature.
update site
set geog = ST_GeogFromText('MULTIPOLYGON(...)')
where id = ...
If you are using the tile filtering feature, you should also update the filter lists in the site_tiles
table.
The downloader should pick the new site definition on the next run.
The Sen2-Agri system supports defining a list of Sentinel-2 and Landsat 8 tiles for each site. This is useful because the uploaded shapefile might overlap the edges of tiles outside the area of interest.
This functionality is not exposed in the web site. To use it, you must first add the site normally, but leave it disabled on creation. You still need to set the season dates and use a suitable shapefile.
After the site is create, you can use the filter_site_download_tiles.py
script to select the desired tiles.
# display the usage information
filter_site_download_tiles.py -h
# disable Landsat 8 for a specific site
filter_site_download_tiles.py -t 2 -s test_czech -e false
# choose specific Sentinel-2 tiles
filter_site_download_tiles.py -t 1 -s test_czech -e true -l "33UVR, 33UWR"
After running the script you can enable the site in the web interface.
The L2A processor uses symbolic links to reduce disk space usage when running MACCS
. This means that the temporary location (/mnt/archive/demmaccs_tmp
) needs to support that feature. Major file systems that don't support symlinks directly are VirtualBox Shared Folders and SMB/CIFS
. See this question for suggestions for handling those.
The paths can be changed in the database, but if you want to store the files somewhere else, like on a larger drive or network volume, it's probably simpler to either mount the drive to /mnt/archive
, or replace those with symlinks:
sudo mv /mnt/archive /mnt/archive_old
sudo ln -s /mnt/large_disk/sen2agri /mnt/archive
If you forgot to do this when installing the system, you should stop the services, replace or re-mount the directories and start the system back up after copying the data.
If you are using SMB
(CIFS
or Windows-like) shares and encounter permission issues while writing to them, you can use something like file_mode=0777,dir_mode=0777,noperm
as mount options, which will make the files world-writable. To improve performance, you can add vers=2.1,cache=loose
. See the mount.cifs
man
page for more details.
A file system supporting symbolic links is required at this time for MACCS, but some workarounds are available.
If you are using VirtualBox you can enable symlink support by running
VBoxManage setextradata VM VBoxInternal2/SharedFoldersEnableSymlinksCreate/share 1
You need to stop and restart the virtual machine after changing this setting; see the VirtualBox manual for more information. On older versions of Windows, you might also need to run VirtualBox with administrator privileges. Please note that performance might be poor when running the system in a virtual machine.
If you are using SMB/CIFS
shares, symlinks can be emulated by storing the metadata in a special file. To enable that, you need to use the mfsymlinks
mount option; see the mount.cifs
man
page for more information.
For versions prior to 1.8, if, for some reason you want to disable the Landsat 8 download for all sites, you can run the following command:
sudo systemctl stop sen2agri-landsat-downloader.{service,timer}
sudo systemctl disable sen2agri-landsat-downloader.timer
To re-enable it back again, run:
sudo systemctl start sen2agri-landsat-downloader.{service,timer}
sudo systemctl enable sen2agri-landsat-downloader.timer
You can also disable the Landsat 8 download for specific sites.
In version 1.8 the L8 download can be disabled from the Edit page of a site.
In versions prior to version 1.8, this can be done by modifying the downloader command line and only works for Sentinel-2 products. You can use the following commands to see and edit the systemd
downloader unit:
systemctl cat sen2agri-sentinel-downloader
sudo systemctl edit sen2agri-sentinel-downloader
In the override file, add a Service
section with an additional argument in the ExecStart
clause, e.g.:
[Service]
ExecStart=/usr/share/sen2agri/sen2agri-downloaders/downloader.py \
-r s2 -s /usr/share/sen2agri/sen2agri-downloaders/apihub.txt \
-l local -i /eodata/Sentinel-2/MSI/L1C
making sure to replace the source path (after -i
). The path needs to be quoted if it contains spaces or other special characters.
After saving the file you need restart the service:
sudo systemctl restart sen2agri-sentinel-downloader
To undo this, simply remove the override file, reload and restart:
sudo rm -rf /etc/systemd/system/sen2agri-sentinel-downloader.service.d
sudo systemctl daemon-reload
sudo systemctl restart sen2agri-sentinel-downloader
Starting with version 1.8, the download is done by the sen2agri-services. In order to configure the services to use local store the following operations should be done:
- Edit the file /usr/share/sen2agri/sen2agri-services/ in order to add/change the following lines:
AWSDataSource.Sentinel2.local_archive_path=/eodata/Sentinel-2/MSI/L1C AWSDataSource.Sentinel2.fetch_mode=4
- Restart the sen2agri-services service using:
sudo systemctl restart sen2agri-services
You might notice that the system starts downloading products three months before the defined season start date. This behaviour is intended and the extra data is used to initialize the L2A processor. MACCS uses a multi-temporal atmospheric correction method and the output quality (of both the surface reflectance valus and the cloud, cloud shadow, snow and water masks) improves with longer time series. A period of three months is estimated to have at least two or three cloud-free dates for each pixel over a given site.
These L2A products are removed by the system once they are no longer needed, to conserve disk space. Besides the MACCS initialization, they are not used for the higher level products.
The system will also try to download products for two months after the end of the season. This is needed for products acquired during the growing season, but processed or published late by ESA or USGS. Products acquired after the season end date will not be downloaded.
SLURM
is a utility that manages resource allocation and job scheduling on a computer cluster. When running an automated or custom (created from the Sen2-Agri web site) job, the system uses it to invoke the processors. While the Sen2-Agri system can run on a single machine, SLURM
allows the processing to span multiple computers.
The Sen2-Agri installer configures SLURM
for a single-machine cluster, but a system administrator can modify these settings as she finds appropriate. The settings include QoS
(Quality of Service) options that can be used to prioritize certain job types. By default the group jobs is set at the installation to 1 but it can be changed to greater value in order to allow parallelization of steps in jobs during processors execution with the following command (for LAI, for example, to increase the number of parallel executed steps to 10):
sudo sacctmgr modify qos qoslai set GrpJobs=10
Describing the SLURM configuration is out of the scope of this document, but more information is available on the SLURM web site.
Note that execution via SLURM is only performed for scheduled and custom jobs.
The system stores its configuration and state in a PostgreSQL database. SLURM
does not support PostgreSQL at this time, so a MariaDB installation is also necessary for its job tracking functionality.
You can use the pgAdmin
application to connect to the database. pgAdmin
is a free and open-source graphical user interface administration tool to manage PostgreSQL databases. To install it, run
sudo yum install pgadmin3
You can then start it from the applications menu or from a terminal, like below:
pgadmin &
disown
Initially, you will need to create a new connection. The host is localhost
if pgadmin
is running on the server, or the server name or address otherwise. You should use the default port (5432
). Use anything for the connection name, admin
as the user name and the sen2agri
password. The database is called sen2agri
.
The base CentOS 7 install has vi
available. You can install joe
or nano
, or vim
as a friendlier version of vi
.
If you get an Error: the MACCS COTS are not installed
error message, the MACCS depedencies were not installed. You should try installing them again.
A common issue is that of pressing y
at the following prompt:
Ready (Y/n) ?
A lower case y
will cancel the installation. You should type the upper case letter Y
.
The core
package will require some additional dependencies. You can install them with:
sudo yum install -y libxslt gd
On some systems you might encounter errors related to geos
or libgdal
during the installation.
One possible cause is that another package (like GRASS GIS
or liblas
) is installed and requires different versions of the common dependencies. In that case, we recommend using a clean CentOS 7
install and not using other GIS software on the same machine.
The other possible reason is that the Sen2-Agri packages need to be re-generated. In that case, you should probably contact us.
You should check the permissions on the /mnt/upload
directory. The web server user (commonly apache
) and the Sen2-Agri system user (sen2agri-service
) need respectively write, and read access.
The easiest way to provide this is to make it world-writable:
# check the permissions
ls -ld /mnt/upload
# mark the directory world-writable
sudo chmod -R 777 /mnt/upload
If you encounter a blank page after uploading a file (e.g. the reference map for a custom L4A
job), you can check the PHP
error log in /var/log/httpd/error_log
.
Most likely, the reason is a configured limit. To increase those, take a look at the post_max_size
and upload_max_filesize
settings in /etc/php.ini
. After changing the configuration file, you must reload it with systemctl reload httpd
.
The SciHub server (used for searching products) has a request size limit. If the site definition is too complex, its WKT
representation might be too long. You should try using a simpler polygon.
The L4
processors are designed to read the full time series at once, for every Sentinel-2 tile that is processed. Since there can be a relatively large number of products and each of them has multiple files, this means that the processors have to open a lot of files. Unfortunately, Linux systems have very strict (small) limits for the number of open files.
These are the settings that influence this behaviour:
-
System-wide open file limit — as configured by the
fs.file-max
sysctl
. This defaults to13 060 355
in CentOS 7, which should be sufficiently large. -
Per-user file descriptor limits — these come in "soft" and "hard" varieties and can be displayed by running
ulimit -n; ulimit -Hn
. The soft limit is applied by default, but processes can raise it up to the value specified by the hard limit. They can be configured in/etc/security/limits.conf
or by placing a file in/etc/security/limits.d
. On CentOS 7 the defaults are1 024
for the soft limit and4 096
for the hard one. -
Per-service file descriptor limit — as applied by a service manager like
systemd
on CentOS 7. For example,samba
has a default limit of16 384
file descriptors. This can be changed by runningsystemctl edit smb
and setting theLimitNOFILE
value in theService
section. -
SLURM
node daemon file descriptor limit — similar to the one above, but defaults to51 200
inSLURM
.
The first point above should not be a problem for the Sen2-Agri system. The installer mitigates the second point by placing the following file:
* hard nofile 500000
* soft nofile 500000
in /etc/security/limits.d/50-open-files.conf
.
The third point may apply under specific situations, e.g. when the input products are located on a Samba share. In that case, you should manually configure the server to increase these limits. A sample override file is the following:
[Service]
LimitNOFILE=500000
The SLURM
limit is configured by a systemd
override file saved in /etc/systemd/system/slurmd.service.d/override.conf
, with the following contents:
[Service]
LimitNOFILE=512000
Note that you might have to reboot the system after changing these values.
Yes, the installer:
- switches SELinux to the permissive mode. There is no technical reason for that, but it has proven somewhat problematic in the past. A determined administrator should be able to re-enable it and make sure that everything is working.
- disables the firewall. The main reason for it is that
SLURM
uses dynamic port allocation. The port range can be restricted, but the Sen2-Agri system does not ship with configuration for this. - increases the open file limits. For more information about this, see this section.
- switches Transparent HugePages to
madvise
mode, effectively disabling them for applications that do not specifically request it. The reason for this is that we noticedTHP
causing a lot of performance problems. This is done by placing the following file under/usr/lib/tmpfiles.d
:
w /sys/kernel/mm/transparent_hugepage/enabled - - - - madvise
These settings are reverted after uninstalling the system.
-
on HP hardware with the
HP Dynamic Smart Array
softwareRAID
controller we've seen poor disk throughput until we changed the system to thethroughput-performance
mode by runningtuned-adm profile throughput-performance
. -
on a dual-socket system we've seen poor performance caused by the automatic
NUMA
balancing feature. It can be disabled by runningecho 0 | sudo tee /proc/sys/kernel/numa_balancing
.
There are, of course, other resources that are better suited to discus Linux performance tuning.
The status of L1
products is stored in the downloader_history
table. You can check it with the following command:
sudo -u postgres psql sen2agri -c \
"select product_name, status_id, status_description \
from downloader_history \
inner join downloader_status \
on downloader_status.id = status_id;"
If you want to e.g. reprocess the failed products, you can change their status:
sudo -u postgres psql sen2agri -c \
"update downloader_history \
set status_id = 2 -- downloaded
where status_id = 6; -- processing_failed"
The status_id
values from the downloader_history
define the status of processing of an L1C product. The possible statuses are defined in the table downloader_status
and they are:
- 1 - "downloading" - the L1C product is currently downloading.
- 2 - "downloaded" - the L1C product was succesfully downloaded but not yet processed by MACCS
- 3 - "failed" - the download of the L1C product failed but it will be retried later
- 4 - "aborted" - the download of the L1C product failed for all retries performed (maximum 3 retries). No other retries will be performed on this product.
- 41 - "download_ignored" - the L1C product contains none of the tiles configured for this site (in case tile filtering is used) and will not be processed by MACCS.
- 5 - "processed" - the L1C product was succesfully processed by MACCS and a valid L2A product corresponding to this L1C is present on the disk and in the
product
table. - 6 - "processing_failed" - the L1C product was successfully downloaded but MACCS failed in processing it (either due to cloud percentage over 90% or due to another error)
- 7 - "processing" - the L1C product was successfully downloaded and MACCS is currently processing it.
You can check the SLURM
status with:
systemctl status slurm{dbd,ctld,d}
Each of the three daemons has a log file in /var/log/slurm
:
sudo tail /var/log/slurm/slurm{dbd,d,}.log
One common issue is MariaDB
not starting after an improper shutdown or reboot. That's apparent from the log files as follows:
[2018-01-03T16:13:08.725] error: mysql_query failed: 145 Table './mysql/proc' is marked as crashed and should be repaired
It can be fixed by running:
$ sudo mysql -p mysql
MariaDB [(mysql)]> repair table proc;
MariaDB [(mysql)]> \q
$ systemctl restart mariadb slurm{dbd,ctld,d}
Starting with version 1.8 a new version of the L3B processor is available. This version is based on the INRA algorithm (S2ToolBox Level 2 products: LAI, FAPAR, FCOVER. Weiss M. et al., 2016) and is creating, beside the LAI and NDVI also the FAPAR and fCover indicators.
The old version is still the by-default on but this new version of the L3B processor can be activated from using the sen2agri-config application and modifying into 1
the key "L3B LAI processor will use INRA algorithm implementation" or alternatively, modify directly the key processor.l3b.lai.use_inra_version
into the config
table.
Additionally, the processor can be configured to generate or not the FAPAR and FCover indicators by setting to 0 or 1 the values L3B LAI processor will produce FAPAR
or L3B LAI processor will produce fCover
(the keys processor.l3b.lai.produce_fapar
respectively processor.l3b.lai.produce_fcover
from the config
table).
If only certain tiles of a site need to be processed by the LAI, then the processor.l3b.lai.tiles_filter
can be set accordingly.
This new implementation is creating much faster the products and also consumes less processor.
Nevertheless, pay attention that this version is working only for Sentinel2 products (L8 is not yet supported but it will be in the version 1.9).
To request assistance, you need an account on the Sentinel-2 for Agriculture portal. The forum is accessible to logged-in users.