[Im4ca-wp1.beta] IM4CA WP1 Romania campaign data and metadata format for ICOS Carbon Portal

[Sara.Defratyka]

Hello everyone, 

Thanks a lot to everyone, who shared their example data and information. As a preparation for data submission, I collected in one palce information obtained from different teams and shared with a person from ICOS CP. You can find them here: 
[ https://sharebox.lsce.ipsl.fr/index.php/s/j68sl8JRPDWroND | https://sharebox.lsce.ipsl.fr/index.php/s/j68sl8JRPDWroND ] 

I hope to arrange another meeting with ICOS CP for this week or the next one, and to come after with some template for in-situ data submission. 

Below you can find a sum-up of what was discussed with ICOS CP in April: 
1) The discussion was dedicated only to data and metadata which supposed to be submitted to ICOS CP. We didn't discuss about storing and sharing raw data within the project. It should be also discussed separately. 

2) We decided to make a distingiush between in situ (CH4 concentration measured in cars and drones and part of aircraft) and remote (imaging by aircrafts and EM27) data, as postprocessing of them takes diffrent amount of time. We came to the conclusion that in-situ data can be submitted as a zip folder of .csv/.txt file and remote data should be wraped in netcdf files. Also, based on the time, remote data won't be ready to be included in this milestone. 

3) although satelite are also involved in the camaping, we didn't discuss it. I believed that during one of weekly camapign meetings, we came to the conclusion, that we don't need to submit them to the ICOS CP, as the are stored separatly on another server and they are quite big. 

4) ICOS CP expects submission of data which can be already used by "end user". Maggie mentioned that metadata and data should be prepared in the way that PhD student at first day of work is able to work on them, casue they are selfe-explanatory. In practise, it means submission of quite polished data and metadata what requires some work from our side. 
It means they won't accept raw data (level 0). They prefere leve 2 data (calibrated and ready to use), but in some circumstances they can accept level 1 data (first autmatic flagging and correction+calibration). There is also possibility to submit later level 3 data (e.g., product based on the data like model results). 

5) What is important, we can resubmit new version of data and metadata once we improve their postprocessing. 

6) We came to the conclusion, that we can use one document with data description per type of measurements and per one submission to facilate the submission process. Again, we can resubmit new version a lot of times. So using car measurments as an example: let's say LSCE data are ready as a first team. We don't need to wait for other teams to have their data ready. The responsible person will just send the data description document plus zip file. And then the next submission can happen when the next team is ready with their data. 

7) We decided to have a data description document where the camapign, data and metadata are described, but also to have short metadata at the beginig of each data file (as a comment starting with #). 

8) We didn't discuss details about name columns yet. But the conclusion is that they should be self-indicating and consistent between files (e.g., CH4_PPM_Picarro shuld mean the same thing using the same type of instrument between different files in one submission). 

9) There is a possibility to prepare a data in a way that they can be previewed on the ICOS CP before downolading. The needed adjustment can be done later. 

10) Embargo- as we are planning to submit the data to ICOS CP quite quickly after campaing, it's obvious we won't calcualte emission, publish paper etc before submission. Should we ask about embargo in this case? Will data submitted with embargo fill the milestone? Also, ICOS CP preferes to not use embargo. 

11) For the data description report, we can start by copy-paste information from the report about equipement readiness from previous deliverable. Working on the data desription document can be started before the campaing. 

12) Finally and probalbyt the most important: as the milestone is not very detailed and deadline is soon, the following proposition was made by ICOS CP: to fill the box, we can just preapre the example of nice data, for example two days of data from one car and from one drone. And then work on the rest of the data and submit full data later, once we are ready. In fact, this solution sounds less stressful. But maybe it is better to ask for postponing the delivery of this milestone and submit full data (what potentially allow us to avoid putting long embargo on the data as well). 

Best, 
Sara 

From: "Sara.Defratyka" <sara.defratyka at lsce.ipsl.fr> 
To: "im4ca-wp1 beta" <im4ca-wp1.beta at listserver.vu.nl> 
Sent: Friday, 24 April, 2026 15:05:35 
Subject: Re: [Im4ca-wp1.beta] IM4CA WP1 Romania campaign data and metadata format for ICOS Carbon Portal 

Hello everyone, 

thanks a lot to those who already sent me their details for data submission to the Carbon Portal and examples of already existing data! The others, please send me needed files and information at the beginnig of the next week, as I wish to send those information to ICOS CP on Tuesday. 

Best, 
Sara 


From: "Sara.Defratyka" <sara.defratyka at lsce.ipsl.fr> 
To: "im4ca-wp1 beta" <im4ca-wp1.beta at listserver.vu.nl> 
Sent: Thursday, 16 April, 2026 12:28:42 
Subject: Re: [Im4ca-wp1.beta] IM4CA WP1 Romania campaign data and metadata format for ICOS Carbon Portal 

Hello everyone, 

based on the last week meeting with ICOS Carbon Portal, regarding data submission for Milestone 1.1 "First EU campaigns datasets", could I ask EACH TEAM to send me details of person who will be invovled from your side for preparation and data submission of data for ICOS Carbon Portal ? Adding those peopel to the system takes some time so it's good to work on it now. I will need: 
- name, 
- emial adress, 
- Orchid number. 

Also, could I ask each team involved in Romania camaping to send me one example of your "ready to publish data" from previous camapings using the same instrument and methodology as you are planning to use during Romania campaing? I will send it further to the person from ICOS CP so they check it and suggest in which direction we should go to make it acceptable for ICOS CP. 

Additionally to the example of data description reports, which I sent some time ago, I'm sending also examples of accepted dataseries on ICOS CP . Please check them and discuss internally within your teams how (and if) your data can be adapted to example format. 

Please come back to me with those information and data until next Thursday, 23.04. 

Best, 
Sara 


From: "Sara.Defratyka" <sara.defratyka at lsce.ipsl.fr> 
To: "im4ca-wp1 beta" <im4ca-wp1.beta at listserver.vu.nl> 
Sent: Monday, 23 March, 2026 10:26:33 
Subject: Re: IM4CA WP1 Romania campaign data and metadata format for ICOS Carbon Portal 

Dear all, 

I'm coming back to the topic of metadata and data, which I started couple of weeks ago. Thanks a lot to Dylan from UU and Stefan from INOE who already volounteer! I think we still need at least one person from aircraft measurement for more effective work. 

In the meantime, could I ask EACH TEAM to fill the list of used instrument durign the camaping, until end of this week 27th of March ? I also added two additioanl columns there (1. in-situ/remote sensor and 2. type of instrument output data), for better preparation of data and metadata format. So, even teams who already filled the form should come back to it and add those details. The spreadsheet is here 
[ https://vu.data.surf.nl/f/222416134 | https://vu.data.surf.nl/f/222416134 ] 

Within the smaller working group, we will have a meeting in the first half of April with Maggie from ICOS CP, to discuss the topic in details. So far I got some more information which everyone should get familiar with and you can find them below. Let me know if you have any questions/concerns and we will discuss it with Maggie. 

Best, 
Sara 



- can submited data be in csv format, in zip package if necessary 


In principle, the data you send in can be in any format, as we usually treat non-ICOS-standard data files as zip archives. The only cases where we demand certain file formats (and variable names etc.) are whenever the data should be previewable at the Carbon Portal – that is only supported for a few very specific types of data files. Burt let’s discuss this more! 





-can we submit 0 level data ( e.,g raw non-calibrated) and if needed submited calibrated updated data later? If we submit updated data, will they get the same reference number and link as initially submited data? 


In general, project data should be Level 2 (fully quality controlled) or Level 3 (elaborated products, such as model outputs) for us to host it. For IM4CA, this is what is mentioned in the project Data Management Plan. Indeed, the Carbon Portal typically does not curate and manage “raw” (Level 0) data that come from non-ICOS sources. But if project data has undergone at least automated basic quality control (as well as temporal aggregation, if applicable) to reach Level 1, and is considered by you to be ready to be shared publicly , we can consider to help you. 


Regarding versions, each object that we upload will be uniquely identified by a basic Handle persistent identifier based on its checksum. If an object is updated, we keep the original version but marl it as being superseded (“deprecated”) by the update. Some datasets will also be assigned a DOI number. For those, we decide on a case-by-case basis whether to reuse the original DOI or assign a new one. QAgain , let’s discuss this in more detail. 





- during campaing we will visit a lot of sites, using a lot of different techniques. What are your recommendation about data harmonizing and submission? Should we submit one file of data per instrument per day? Or maybe divide it based on visited sources? What harmonization approach should we take for data measurement by different instruments which are analysed togheter (e.g., CH4 instrument, GPS and anemomenter )? 


Here there is no “one size fits all” approach; as long as the approach is well documented there is room for many different solutions – also within a campaign, as the various types of observations and instrumentation must be considered separately. My thinking is that there should be a focus on what any end users of the data would want, and decide the strategy and structure based on that. 





- what is recomend denotation for flagging data? Should flagged data stay in the submitted dataset? 


For Level 1 and Level 2 data, it is always preferable to include Flag values. 





- Should names of variables be consinstent between different instrument? I assume sometimes it cannot be possible to achieve 


That is not necessary, but variable and/or column names inside data files should as a rule be at least indicative of what they contain. (Avoid “variable1”, “variable2” or using e.g. “ch4” to mean different things in different types of data (i.e. flux in one file and mole fraction in another). Again, this should be clearly described in the accompanying documentation! 





- Can different data sets be submitted in separate time by different people (e.g., data colelction per involved institution)? 


Yes, sure. We have to discuss the procedures here, and divide responsibilities wisely. But since I would propose that daily L0 data are not sent to the Carbon Portal but rather be stored in your Research Drive, the would leave a smaller number of L1 and L2 objects – and if there aren’t too many of these, I can take care of the actual uploads as long as you provide the necessary metadata. Let’s talk more about this as well! 


From: "Sara.Defratyka" <sara.defratyka at lsce.ipsl.fr> 
To: "im4ca-wp1 beta" <im4ca-wp1.beta at listserver.vu.nl> 
Sent: Wednesday, 11 March, 2026 13:35:21 
Subject: IM4CA WP1 Romania campaign data and metadata format for ICOS Carbon Portal 



Dear all, 



As you may know, Milestone 1.1 (First EU Campaigns Datasets Published) is due one month after the Romania campaign . To facilitate progress on this task, I propose that we start working on the data and metadata format now, with the goal of finalizing the template before the campaign. This should help streamline the data submission process to the ICOS Carbon Portal (ICOS CP) after the campaign. 



I have already contacted ICOS CP and obtained two example reports as good practice references. You can find these attached, along with the IM4CA Data Management Plan and Deliverable 1.1 , which can serve as a starting point for data description in Milestone 1.1 . These documents are also available on the shared drive of WP1 in the "Milestones" folder. 



The aim is to create a template that is as user-friendly as possible for us while ensuring it meets ICOS CP’s requirements. To begin, I propose forming a small working group with one representative per measurement technique (car, drone, aircraft, stationary EM27, stationary CRDS). Over time, we may need to expand the group—for example, to include one representative per institution —but we’ll assess this as we progress. 



Once we make significant progress within this small group (with support from ICOS CP ), I will update the rest of WP1 to keep everyone informed while minimizing unnecessary emails and meetings. 



If you are interested in joining this working group, please reply to this email. I understand that everyone is very busy, so I will keep the workload on this part of the Romania campaign as light as possible. 



Best, 

Sara 

-- 
Dr Sara Defratyka 
orcid.org/0000-0002-9488-4518 
LSCE - UVSQ 
Bât 714 P1115 
Orme des Merisiers 
91191 Gif-sur-Yvette 



[Text File:ATT00001] 


[Text File:ATT00001] 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://listserver.vu.nl/pipermail/im4ca-wp1.beta/attachments/20260518/f0392017/attachment-0001.html>