2024-12-02T10:08:15Z INFO __main__ 305:
PIPELINE VERSION
Version: 1.5.0
Common instrument config version: v0.1.5
PROCESSING VERSIONS
Prod: Ephemeris
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/ANC/aspect.py
Vers: 2
Prod: MiniReport
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level0/housekeepingL0.py
Vers: 2
Prod: MaxiReport
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level0/housekeepingL0.py
Vers: 2
Prod: LightCurve
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level0/quicklookL0.py
Vers: 2
Prod: Background
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level0/quicklookL0.py
Vers: 2
Prod: Spectra
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level0/quicklookL0.py
Vers: 2
Prod: Variance
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level0/quicklookL0.py
Vers: 2
Prod: FlareFlag
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level0/quicklookL0.py
Vers: 2
Prod: EnergyCalibration
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level0/quicklookL0.py
Vers: 2
Prod: TMStatusFlareList
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level0/quicklookL0.py
Vers: 2
Prod: RawPixelData
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level0/scienceL0.py
Vers: 2
Prod: CompressedPixelData
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level0/scienceL0.py
Vers: 2
Prod: SummedPixelData
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level0/scienceL0.py
Vers: 2
Prod: Visibility
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level0/scienceL0.py
Vers: 2
Prod: Spectrogram
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level0/scienceL0.py
Vers: 2
Prod: Aspect
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level0/scienceL0.py
Vers: 2
Prod: MiniReport
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level1/housekeepingL1.py
Vers: 2
Prod: MaxiReport
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level1/housekeepingL1.py
Vers: 2
Prod: LightCurve
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level1/quicklookL1.py
Vers: 2
Prod: Background
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level1/quicklookL1.py
Vers: 2
Prod: Spectra
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level1/quicklookL1.py
Vers: 2
Prod: Variance
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level1/quicklookL1.py
Vers: 2
Prod: FlareFlag
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level1/quicklookL1.py
Vers: 2
Prod: EnergyCalibration
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level1/quicklookL1.py
Vers: 2
Prod: TMStatusFlareList
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level1/quicklookL1.py
Vers: 2
Prod: RawPixelData
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level1/scienceL1.py
Vers: 2
Prod: CompressedPixelData
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level1/scienceL1.py
Vers: 2
Prod: SummedPixelData
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level1/scienceL1.py
Vers: 2
Prod: Visibility
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level1/scienceL1.py
Vers: 2
Prod: Spectrogram
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level1/scienceL1.py
Vers: 2
Prod: Aspect
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level1/scienceL1.py
Vers: 2
Prod: MiniReport
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level2/housekeepingL2.py
Vers: 2
Prod: MaxiReport
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level2/housekeepingL2.py
Vers: 2
Prod: LightCurve
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level2/quicklookL2.py
Vers: 2
Prod: Background
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level2/quicklookL2.py
Vers: 2
Prod: Spectra
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level2/quicklookL2.py
Vers: 2
Prod: Variance
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level2/quicklookL2.py
Vers: 2
Prod: FlareFlag
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level2/quicklookL2.py
Vers: 2
Prod: EnergyCalibration
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level2/quicklookL2.py
Vers: 2
Prod: TMStatusFlareList
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/level2/quicklookL2.py
Vers: 2
Prod: LevelB
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/levelb/binary.py
Vers: 2
Prod: LightCurve
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/lowlatency/quicklookLL.py
Vers: 2
Prod: FlareFlag
File: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/products/lowlatency/quicklookLL.py
Vers: 2
2024-12-02T10:08:15Z INFO __main__ 270:
CONFIG
[Paths]
tm_archive = /data/stix/SOLSOC/from_edds/tm/processing
fits_archive = /data/stix/out/fits_v1.2.0
spice_kernels = /data/stix/spice/kernels
soop_files = /data/stix/SOLSOC/from_soc
[Logging]
stop_on_error = False
[IDLBridge]
enabled = True
gsw_path = /opt/STIX-GSW
batchsize = 30
[Pipeline]
status_server_port = 12388
parallel_batchsize_l0 = 100
parallel_batchsize_l1 = 100
parallel_batchsize_l2 = 100
log_level = INFO
log_dir = /data/stix/out/fits_v1.2.0/logging
error_mail_send = True
error_mail_receivers = nicky.hochmuth@ateleris.ch, shane.maloney@dias.ie
error_mail_smpt_host = lmailer.fhnw.ch
error_mail_smpt_port = 25
error_mail_sender = nicky.hochmuth@fhnw.ch
start_with_unprocessed = True
history_db_file = /home/stixcore/soar_pub/processing_history.sqlite
[Publish]
db_file = /home/stixcore/soar_pub/published.sqlite
include_levels = L0, L1, ANC
waiting_period = 21d
target_dir = /home/stixcore/from_pub099
report_mail_receivers = nicky.hochmuth@ateleris.ch, shane.maloney@dias.ie
report_mail_send = True
same_esa_name_dir = /data/stix/out/esa_conflicts
rid_lut_file = ./stixcore/data/publish/rid_lut.csv
rid_lut_file_update_url = https://datacenter.stix.i4ds.net/api/bsd/info/
batch_size = 20000
fits_dir = /data/stix/out/fits_v1.2.0
target_host = pub026.cs.technik.fhnw.ch
supplement_report = /home/stixcore/soar_pub/supplement_report.csv
include_products = ql,hk,sci,cal
log_dir = /data/stix/out/fits_v1.2.0/logging
[SOOP]
endpoint = https://solarorbiter.esac.esa.int/soopkitchen/api
user = smaloney
password = xxxx
soop_files_download = ./stixcore/data/soop
2024-12-02T10:08:15Z INFO __main__ 301:
SINGLETONS
SOOPManager: /data/stix/SOLSOC/from_soc
RidLutManager: file: stixcore/data/publish/rid_lut.csv update: True size: 33663
SPICE: [(PosixPath('/data/stix/spice/kernels/mk/solo_ANC_soc-pred-mk_V111_20241127_004.tm'), 'pred', datetime.datetime(2024, 11, 27, 0, 0)), (PosixPath('/data/stix/spice/kernels/mk/solo_ANC_soc-flown-mk_V111_20241130_002.tm'), 'flown', datetime.datetime(2024, 11, 30, 0, 0))]
IDBManager: /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/data/idb
Versions:
[{'label': '2.26.36', 'path': '/home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/data/idb/v2.26.36', 'version': ['2', '26', '36']}, {'label': '2.26.32', 'path': '/home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/data/idb/v2.26.32', 'version': ['2', '26', '32']}, {'label': '2.26.34', 'path': '/home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/data/idb/v2.26.34', 'version': ['2', '26', '34']}, {'label': '2.26.38', 'path': '/home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/data/idb/v2.26.38', 'version': ['2', '26', '38']}, {'label': '2.26.33', 'path': '/home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/data/idb/v2.26.33', 'version': ['2', '26', '33']}, {'label': '2.26.31', 'path': '/home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/data/idb/v2.26.31', 'version': ['2', '26', '31']}, {'label': '2.26.35', 'path': '/home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/data/idb/v2.26.35', 'version': ['2', '26', '35']}]
Force version: None
History:
IntervalTree([Interval(0.0, 640198039.0, '2.26.32'), Interval(640198039.0, 640265396.0, '2.26.31'), Interval(640265396.0, 642159993.0, '2.26.32'), Interval(642159993.0, 643123725.0, '2.26.33'), Interval(643123725.0, 677774249.0, '2.26.34'), Interval(677774249.0, 692380776.0, '2.26.35'), Interval(692380776.0, 761270182.0, '2.26.36'), Interval(761270182.0, 4294967295.0, '2.26.38')])
2024-12-02T10:08:15Z INFO stixcore.io.RidLutManager 158: download publish lut file: https://datacenter.stix.i4ds.net/api/bsd/info/20241124/20241224
2024-12-02T10:08:15Z INFO stixcore.io.RidLutManager 167: found 2 entries
2024-12-02T10:08:17Z INFO stixcore.io.RidLutManager 176: write total 33663 entries to local storage
2024-12-02T10:08:17Z INFO stixcore.processing.TMTCtoLB 30: Processing file: SOCPacketFile('/data/stix/SOLSOC/from_edds/tm/processing/TM_hourly_req_BatchRequest.PktTmRaw.SOL.0.2024.184.14.26.05.512.pjLt@2024.335.18.00.04.643.1.xml')
2024-12-02T10:08:17Z INFO __main__ 495: Opened file: /data/stix/SOLSOC/from_edds/tm/processing/TM_hourly_req_BatchRequest.PktTmRaw.SOL.0.2024.184.14.26.05.512.pjLt@2024.335.18.00.04.643.1.xml
2024-12-02T10:08:17Z INFO __main__ 490: Closed read file: /data/stix/SOLSOC/from_edds/tm/processing/TM_hourly_req_BatchRequest.PktTmRaw.SOL.0.2024.184.14.26.05.512.pjLt@2024.335.18.00.04.643.1.xml
2024-12-02T10:08:17Z INFO __main__ 495: Opened file: /data/stix/SOLSOC/from_edds/tm/processing/TM_hourly_req_BatchRequest.PktTmRaw.SOL.0.2024.184.14.26.05.512.pjLt@2024.335.18.00.04.643.1.xml
2024-12-02T10:08:17Z INFO __main__ 490: Closed read file: /data/stix/SOLSOC/from_edds/tm/processing/TM_hourly_req_BatchRequest.PktTmRaw.SOL.0.2024.184.14.26.05.512.pjLt@2024.335.18.00.04.643.1.xml
2024-12-02T10:08:17Z INFO stixcore.io.fits.processors 448: Fits file solo_LB_stix-21-6-42_0000000000-9999999999_V02_2410014254-00000.fits exists appending data
2024-12-02T10:08:17Z INFO stixcore.io.fits.processors 448: Fits file solo_LB_stix-21-6-21_0000000000-9999999999_V02_2409279396-64721.fits exists appending data
2024-12-02T10:08:17Z INFO stixcore.io.fits.processors 479: Writing fits file to /data/stix/out/fits_v1.2.0/LB/21/6/42/solo_LB_stix-21-6-42_0000000000-9999999999_V02_2410014254-00000.fits
2024-12-02T10:08:17Z INFO stixcore.io.fits.processors 479: Writing fits file to /data/stix/out/fits_v1.2.0/LB/21/6/21/solo_LB_stix-21-6-21_0000000000-9999999999_V02_2409279396-64721.fits
2024-12-02T10:08:17Z INFO __main__ 224: generated LB files:
{PosixPath('/data/stix/out/fits_v1.2.0/LB/21/6/21/solo_LB_stix-21-6-21_0000000000-9999999999_V02_2409279396-64721.fits'),
PosixPath('/data/stix/out/fits_v1.2.0/LB/21/6/42/solo_LB_stix-21-6-42_0000000000-9999999999_V02_2410014254-00000.fits')}
2024-12-02T10:08:17Z INFO stixcore.idb.idb 522: IDB loaded from /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/data/idb/v2.26.38/idb.sqlite
2024-12-02T10:08:17Z INFO stixcore.idb.idb 522: IDB loaded from /home/stixcore/STIXCore/venv/lib/python3.9/site-packages/stixcore/data/idb/v2.26.38/idb.sqlite
2024-12-02T10:08:18Z INFO stixcore.ephemeris.manager 198: LOADING NEW META KERNEL: /data/stix/spice/kernels/mk/solo_ANC_soc-pred-mk_V111_20241127_004.tm
2024-12-02T10:08:18Z INFO stixcore.ephemeris.manager 198: LOADING NEW META KERNEL: /data/stix/spice/kernels/mk/solo_ANC_soc-pred-mk_V111_20241127_004.tm
2024-12-02T10:08:18Z INFO stixcore.ephemeris.manager 198: LOADING NEW META KERNEL: /data/stix/spice/kernels/mk/solo_ANC_soc-flown-mk_V111_20241130_002.tm
2024-12-02T10:08:18Z INFO stixcore.io.RidLutManager 178: read rid-lut from stixcore/data/publish/rid_lut.csv
2024-12-02T10:08:18Z INFO stixcore.ephemeris.manager 198: LOADING NEW META KERNEL: /data/stix/spice/kernels/mk/solo_ANC_soc-flown-mk_V111_20241130_002.tm
2024-12-02T10:08:18Z INFO stixcore.io.RidLutManager 178: read rid-lut from stixcore/data/publish/rid_lut.csv
2024-12-02T10:08:19Z WARNING stixcore.products.levelb.binary 322: Incomplete sequence <LevelB LB, 21, 6, 21 785635010:14478-785635936:60184>
sequence_count sequence_flag scet_coarse
-------------- ------------- -----------
13902 1 785635010
13903 0 785635028
13904 0 785635045
13905 0 785635064
13906 0 785635079
13907 0 785635093
13908 0 785635107
13909 0 785635114
13910 0 785635114
13911 0 785635126
13912 0 785635132
13913 0 785635144
13914 0 785635154
13915 0 785635159
13916 0 785635163
13917 0 785635173
13918 0 785635177
13919 0 785635177
13920 0 785635188
13921 0 785635192
13922 0 785635196
13923 0 785635206
13924 0 785635210
13925 0 785635220
13926 0 785635224
13927 0 785635224
13928 0 785635227
13929 0 785635237
13930 0 785635240
13931 0 785635244
13932 0 785635254
13933 0 785635258
13934 0 785635268
13935 0 785635272
13936 0 785635272
13937 0 785635282
13938 0 785635287
13939 0 785635298
13940 0 785635304
13941 0 785635314
13942 0 785635320
13943 0 785635337
13944 0 785635337
13945 0 785635366
13946 0 785635397
13947 0 785635436
13948 0 785635468
13949 0 785635500
13950 0 785635541
13951 0 785635574
13952 0 785635604
13953 0 785635605
13954 0 785635633
13955 0 785635656
13956 0 785635683
13957 0 785635703
13958 0 785635726
13959 0 785635736
13960 0 785635749
13961 0 785635750
13962 0 785635762
13963 0 785635775
13964 0 785635780
13965 0 785635793
13966 0 785635809
13967 0 785635827
13968 0 785635852
13969 0 785635869
13970 0 785635870
13971 0 785635886
13972 0 785635899
13973 0 785635905
13974 0 785635916
13975 0 785635921
13976 0 785635931
13977 0 785635936
13978 0 785635936
2024-12-02T10:08:19Z WARNING stixcore.products.levelb.binary 322: Incomplete sequence <LevelB LB, 21, 6, 42 786300269:22350-786300594:23093>
sequence_count sequence_flag scet_coarse
-------------- ------------- -----------
1303 1 786300269
1304 0 786300278
1305 0 786300286
1306 0 786300295
1307 0 786300304
1308 0 786300313
1309 0 786300322
1310 0 786300330
1311 0 786300342
1312 0 786300351
1313 0 786300360
1314 0 786300368
1315 0 786300377
1316 0 786300385
1317 0 786300394
1318 0 786300403
1319 0 786300411
1320 0 786300420
1321 0 786300429
1322 0 786300438
1323 0 786300446
1324 0 786300455
1325 0 786300464
1326 0 786300473
1327 0 786300479
1328 0 786300485
1329 0 786300494
1330 0 786300502
1332 0 786300514
1333 0 786300523
1334 0 786300532
1335 0 786300541
1336 0 786300550
1337 0 786300559
1338 0 786300567
1339 0 786300576
1340 0 786300585
1341 0 786300594
2024-12-02T10:08:19Z INFO __main__ 228: generated L0 files:
[]
2024-12-02T10:08:19Z INFO __main__ 232: generated L1 files:
[]
2024-12-02T10:08:19Z INFO __main__ 236: generated L2 files:
[]