{ "cells": [ { "cell_type": "markdown", "id": "c2bdb999-2e16-4160-96e6-906c45ff8b41", "metadata": {}, "source": [ "Formatting Your Data for Eureka! Light Curve Extraction and Fitting\n", "====================================================================" ] }, { "cell_type": "markdown", "id": "1942629d-7765-4214-ba8d-6388eab5682a", "metadata": {}, "source": [ "This notebook will give a quick example of how to format an arbitrary dataset for input into Eureka! to make use of its light curve extraction and fitting routines (i.e. Eureka! Stage 4 and Stage 5, respectively). At the moment, this notebook is written specifically in the context of understanding and mimicking the Stage 3 Eureka! output data, which contains extracted 1D spectra." ] }, { "cell_type": "markdown", "id": "c13d3d3a-5d8f-45c1-8339-f2376c679f04", "metadata": {}, "source": [ "The Eureka! Data Format\n", "------------------\n", "To begin, let's load a Eureka! file to see how things *should* be formatted. " ] }, { "cell_type": "code", "execution_count": 1, "id": "6727cc7c-9a2a-4386-85fa-5516ef6c34ad", "metadata": {}, "outputs": [], "source": [ "base_dir = '/Users/acarter/Documents/TRANSITS/EUREKA/ALTERNATE_INPUTS/NIRCam_TESTING/'\n", "eureka_file = base_dir + 'Stage3/S3_2024-01-10_nircam_template_run1/ap8_bg8/S3_nircam_template_ap8_bg8_SpecData.h5'" ] }, { "cell_type": "markdown", "id": "93f6909a-b620-45e3-a642-acca5c844c7b", "metadata": {}, "source": [ "You'll see that this is a `.h5` file. Many Eureka! outputs are saved in this format, and more specifically as [Xarray](https://docs.xarray.dev/en/latest/index.html) objects. This is a data format that allows for storing complex datasets, and requires specific libraries to be read into Python. There are multiple options, but we'll use the [Astraeus](https://kevin218.github.io/Astraeus/) library created by Kevin Stevenson." ] }, { "cell_type": "code", "execution_count": 2, "id": "7a1e1971-3ef2-41dc-af2d-a2d4695c21df", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Finished loading parameters from /Users/acarter/Documents/TRANSITS/EUREKA/ALTERNATE_INPUTS/NIRCam_TESTING/Stage3/S3_2024-01-10_nircam_template_run1/ap8_bg8/S3_nircam_template_ap8_bg8_SpecData.h5\n" ] } ], "source": [ "from astraeus.xarrayIO import readXR\n", "eureka_data = readXR(eureka_file)" ] }, { "cell_type": "markdown", "id": "78059609-a820-446b-a917-a63429268b92", "metadata": {}, "source": [ "Our file was succesfully loaded, so let's take a look at what's inside:" ] }, { "cell_type": "code", "execution_count": 3, "id": "aa763276-e329-4c4d-9eaa-51a3bfda29fe", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Dimensions: (time: 128, y: 59, x: 1600)\n", "Coordinates:\n", " * time (time) float64 5.969e+04 5.969e+04 ... 5.969e+04 5.969e+04\n", " * y (y) int64 5 6 7 8 9 10 11 12 13 ... 55 56 57 58 59 60 61 62 63\n", " * x (x) int64 100 101 102 103 104 105 ... 1695 1696 1697 1698 1699\n", "Data variables:\n", " wave_1d (x) float32 ...\n", " medflux (y, x) float64 ...\n", " centroid_y (time) float64 ...\n", " centroid_sy (time) float64 ...\n", " stdspec (time, x) float32 ...\n", " stdvar (time, x) float32 ...\n", " optspec (time, x) float64 ...\n", " opterr (time, x) float64 ...\n", " optmask (time, x) bool ...\n", "Attributes: (12/74)\n", " ncpu: 1\n", " nfiles: 1\n", " max_memory: 0.5\n", " indep_batches: False\n", " suffix: calints\n", " calibrated_spectra: False\n", " ... ...\n", " bg_y2: 38\n", " bg_y1: 21\n", " save_fluxdata: False\n", " citations: ['astraeus', 'astropy', 'crds', 'eureka', 'h5py', ...\n", " bibliography: ['@misc{astraeus,\\n author = {{Stevenso...\n", " filename_S3_SpecData: /Users/acarter/Documents/TRANSITS/EUREKA/ALTERNATE...\n" ] } ], "source": [ "print(eureka_data)" ] }, { "cell_type": "markdown", "id": "ded0cbed-7b3b-42f4-a6f9-a6f2aae2908d", "metadata": {}, "source": [ "
\n", "Phew! There's a lot going on here, so let's take it step by step:\n", "
 \n", "\n", "`Dimensions:` This tells us the *potential* dimensionality of various arrays within the dataset. In this case there are 3 potential dimensions: time (the time steps for each measurement), y (the detector y pixels), and x (the detector x pixels). \n", " \n", "\n", "`Coordinates:` This tells us the specific values for the coordinates that correspond to the previously mentioned `Dimensions`. Here we can see the time is recorded in MJD as a float, and the x- and y-pixel values are recorded as integers. Note that the x and y coordinates don't have to start at zero, as the initial detector array was cropped during spectral extraction. \n", " \n", "\n", "`Data variables:` This tells us the various data arrays that are saved in the dataset. We won't go through them all here, but notice how their dimensionality corresponds to the `Dimensions`. For example, `wave_1d` is a data variable of the x-pixel dimension, whereas `optspec` is a variable of the time *and* x-pixel dimension. This makes sense, as each x-pixel should have a corresponding wavelength (in a case like this, where the x-axis is the dispersion axis). Similarly, all of our extracted 1D spectra, `optspec`, should have a corresponding time at which each spectrum was measured, and a corresponding x-pixel for each pixel column within the extracted region. \n", " \n", "\n", "`Attributes:` Finally, this tells us a range of metadata attributes corresponding to various settings and outputs from the Eureka! processing. For example, the y-pixel limits at which the background flux was estimated, the output filename of the data, and more. \n", " \n", "\n", "Remember this is just the default output from Eureka! for its Stage 3 spectral extraction. When generating a custom file for input into Stage 4, we don't necessarily need to define all of these variables and attributes." ] }, { "cell_type": "markdown", "id": "acb53bfb-650a-45d8-9f3f-e02156b88ea2", "metadata": {}, "source": [ "Mimicking The Eureka! Format\n", "----------------------------\n", "Okay, so let's say you want to create your own file to feed into Eureka! so that you can make use of it's light curve extraction and fitting functionality. Assuming your data are already in the form of an extracted spectral time series, here is what you will need:\n", " \n", "\n", "**Required**\n", "\n", "* The time steps for each of your spectra (`time`), and units.\n", "* The wavelength array for the spectra (`wave_1d`), and units.\n", "* The extracted spectrum at each time step (`optspec`), and units.\n", "* The error on the extracted spectrum at each time step (`opterr`), and units.\n", "\n", "**Optional**\n", "\n", "* A 2D mask for specific points to ignore in the spectral time series (`optmask`).\n", "* The shift in the spectral trace centroid and width over time (`centroid_y` and `centroid_sy`, respectively).\n", "* Various other variables and attributes.\n", " \n", "\n", "Using this as a rough guide, let's build a mock dataset." ] }, { "cell_type": "code", "execution_count": 4, "id": "f8357a19-f96f-40ad-959b-53aec69bd53d", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", "time = np.linspace(59694.0, 59694.1, 100)\n", "time_units = 'MJD'\n", "wave_1d = np.linspace(1.0, 5.0, 1001)\n", "x = np.arange(len(wave_1d))\n", "wave_units = 'microns'\n", "optspec = np.ones((len(time), len(wave_1d)))\n", "opterr = np.ones((len(time), len(wave_1d)))*0.01\n", "flux_units = 'e-'" ] }, { "cell_type": "markdown", "id": "6e1573b1-6745-4b23-86f0-740d6e95d00a", "metadata": {}, "source": [ "Note that these values are entirely made up for the purpose of this notebook - you should be setting these variables to *your* data values!\n", " \n", "\n", "To build the Xarray, we will again use the [Astraeus](https://kevin218.github.io/Astraeus/) library. Let's start by initialising a dataset:" ] }, { "cell_type": "code", "execution_count": 5, "id": "e091de62-12b0-4f56-b29f-88dfd52b6014", "metadata": {}, "outputs": [], "source": [ "from astraeus.xarrayIO import makeDataset\n", "\n", "custom_dataset = makeDataset()" ] }, { "cell_type": "markdown", "id": "13520c0f-d3ab-4b31-a94d-1e4e960d3b45", "metadata": {}, "source": [ "Now we can assign all of the arrays from above to the correct Xarray DataArray format: " ] }, { "cell_type": "code", "execution_count": 6, "id": "a0b8a690-431b-401b-ba82-e3fbf5cb75af", "metadata": {}, "outputs": [], "source": [ "custom_dataset['time'] = time\n", "custom_dataset['time'].attrs['time_units'] = time_units\n", "custom_dataset['x'] = x\n", "custom_dataset['optspec'] = (['time', 'x'], optspec)\n", "custom_dataset['optspec'].attrs['flux_units'] = flux_units\n", "custom_dataset['optspec'].attrs['time_units'] = time_units\n", "custom_dataset['optspec'].attrs['wave_units'] = wave_units\n", "custom_dataset['opterr'] = (['time', 'x'], opterr)\n", "custom_dataset['opterr'].attrs['flux_units'] = flux_units\n", "custom_dataset['opterr'].attrs['time_units'] = time_units\n", "custom_dataset['opterr'].attrs['wave_units'] = wave_units\n", "custom_dataset['wave_1d'] = (['x'], wave_1d)\n", "custom_dataset['wave_1d'].attrs['wave_units'] = wave_units" ] }, { "cell_type": "markdown", "id": "01cb4e9b-ef7d-4791-af6f-84bc22d36b72", "metadata": {}, "source": [ "As you can see it's relatively straightforward to assign everything. If you wanted to expand what to include in the custom_dataset, simply assign it to a new dictionary key in `custom_dataset`, but ensure it matches a variable that Eureka! will be looking for. Now, let's see what the dataset looks like:" ] }, { "cell_type": "code", "execution_count": 7, "id": "2a30c145-1645-4cd2-bfcd-f2591b51d724", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Dimensions: (time: 100, x: 1001)\n", "Coordinates:\n", " * time (time) float64 5.969e+04 5.969e+04 ... 5.969e+04 5.969e+04\n", " * x (x) int64 0 1 2 3 4 5 6 7 8 ... 993 994 995 996 997 998 999 1000\n", "Data variables:\n", " optspec (time, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 ... 1.0 1.0 1.0 1.0 1.0\n", " opterr (time, x) float64 0.01 0.01 0.01 0.01 0.01 ... 0.01 0.01 0.01 0.01\n", " wave_1d (x) float64 1.0 1.004 1.008 1.012 1.016 ... 4.988 4.992 4.996 5.0\n" ] } ], "source": [ "print(custom_dataset)" ] }, { "cell_type": "markdown", "id": "767898dd-e012-4ef8-ad63-334de90ed710", "metadata": {}, "source": [ "
\n", "\n", "All of the data is now in the correct format, and the variable names match those from the actual Eureka! file. The only thing that remains is to assign any relevant `Attributes`. Advanced users may wish to assign more, but for most use cases, only two are mandatory - the instrument the data was obtained with, and a flag to let Eureka! know it's looking at a custom dataset." ] }, { "cell_type": "code", "execution_count": 8, "id": "420213bb-b270-4a4d-9493-c5883172dced", "metadata": {}, "outputs": [], "source": [ "# Specify instrument, options are: 'nircam', 'miri', 'nirspec', 'nircam', 'wfc3'\n", "custom_dataset.attrs['inst'] = 'nircam'\n", "# Specify data_format, must be set to 'custom'\n", "custom_dataset.attrs['data_format'] = 'custom'" ] }, { "cell_type": "markdown", "id": "f334c30e-c403-4ff9-8c0a-5a0620907dc4", "metadata": {}, "source": [ "Finally, we can use Astraeus once more to save the dataset to a Eureka! compatible file.\n", " \n", "\n", "
\n", "\n", "Important!\n", " \n", "\n", "To mimic a Stage 3 file, your custom file _**must**_ be named correctly. Specifically, it must follow the format:\n", "\n", "\n", "S3_{ecf_event_label}{other_file_text}_SpecData.h5 \n", "
 \n", "\n", "You may replace `other_file_text` with any text you like. The `ecf_event_label` is connected to the filename of the Stage 4 ECF file you are planning to use; if you are unfamiliar with Eureka! event labels, please see the quickstart description **[here](https://eurekadocs.readthedocs.io/en/latest/quickstart.html#customise-the-demo-files)**.\n", "\n", "
" ] }, { "cell_type": "code", "execution_count": 9, "id": "eb27001c-3ef3-422f-a73e-878095e67af2", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Finished writing to /Users/acarter/Documents/TRANSITS/EUREKA/ALTERNATE_INPUTS/NIRCam_TESTING/Custom_Input/S3_custom_dataset_SpecData.h5\n" ] }, { "data": { "text/plain": [ "True" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from astraeus.xarrayIO import writeXR\n", "import os\n", "\n", "# Make sure we have made the directory to save the file to\n", "save_dir = base_dir+'Custom_Input/'\n", "if not os.path.exists(save_dir):\n", " os.makedirs(save_dir)\n", "save_file = save_dir+'S3_custom_dataset_SpecData.h5' \n", "\n", "# Save the file\n", "writeXR(save_file, custom_dataset, verbose=True)" ] }, { "cell_type": "markdown", "id": "53605ea7-ab61-4df0-9d10-25ac06ebcbd6", "metadata": {}, "source": [ "Congratulations! You've learned everything you need to make your own Eureka! compatible dataset. Feel free to repurpose or modify any of the cells in this notebook to suit your actual data. \n", "\n", "If you were hoping to import some photometric data, please read the section below.\n", "\n", "If this made sense, but you are still unfamiliar with how to use Eureka!, please see the **[Quickstart Guide](https://eurekadocs.readthedocs.io/en/latest/quickstart.html)**.\n", "\n", "Finally, if you encounter any issues or bugs while importing your own data, please contact the Eureka! team directly by making a **[GitHub Issue](https://github.com/kevin218/Eureka/issues)**." ] }, { "cell_type": "markdown", "id": "3af689ce-a4f6-4506-8ac5-c7a9caa740fd", "metadata": {}, "source": [ "An Aside On Photometry\n", "----------------------\n", "\n", "If you would like to import a photometry dataset, some slight adjustments must be made. Let's take a look at what a Eureka! photometry Stage 3 output looks like:" ] }, { "cell_type": "code", "execution_count": 10, "id": "9ffd0089-4cd3-4ab3-a039-68bb2af8b6c4", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Finished loading parameters from /Users/acarter/Documents/TRANSITS/EUREKA/ALTERNATE_INPUTS/NIRCam_TESTING/Stage3_Photom/S3_2024-04-30_nircam_template_run2/ap60_bg70_90/S3_nircam_template_ap60_bg70_90_SpecData.h5\n", "\n", "Dimensions: (time: 119, y: 252, x: 1024)\n", "Coordinates:\n", " * time (time) float64 5.978e+04 5.978e+04 ... 5.978e+04 5.978e+04\n", " * y (y) int64 4 5 6 7 8 9 10 11 ... 248 249 250 251 252 253 254 255\n", " * x (x) int64 512 513 514 515 516 517 ... 1531 1532 1533 1534 1535\n", "Data variables: (12/14)\n", " wave_1d (x) float32 ...\n", " centroid_x (time) float64 ...\n", " centroid_y (time) float64 ...\n", " centroid_sx (time) float64 ...\n", " centroid_sy (time) float64 ...\n", " aplev (time) float64 ...\n", " ... ...\n", " skylev (time) float64 ...\n", " skyerr (time) float64 ...\n", " nskypix (time) float64 ...\n", " nskyideal (time) float64 ...\n", " status (time) float64 ...\n", " betaper (time) float64 ...\n", "Attributes: (12/68)\n", " ncpu: 1\n", " nfiles: 1\n", " max_memory: 0.5\n", " suffix: calints\n", " photometry: True\n", " convert_to_e: True\n", " ... ...\n", " skyout: 90\n", " gainfile: /Users/acarter/crds_cache/references/jwst/nircam/j...\n", " save_fluxdata: False\n", " citations: ['astraeus', 'astropy', 'crds', 'eureka', 'h5py', ...\n", " bibliography: ['@misc{astraeus,\\n author = {{Stevenso...\n", " filename_S3_SpecData: /Users/acarter/Documents/TRANSITS/EUREKA/ALTERNATE...\n" ] } ], "source": [ "photom_file = base_dir + 'Stage3_Photom/S3_2024-04-30_nircam_template_run2/ap60_bg70_90/S3_nircam_template_ap60_bg70_90_SpecData.h5'\n", "photom_data = readXR(photom_file)\n", "print(photom_data)" ] }, { "cell_type": "markdown", "id": "29fe58ce-026d-436d-b079-5ac80d81d02f", "metadata": {}, "source": [ "
\n", "\n", "As you can see the general structure is similar, but some of the data variables have changed. Specifically, the `optspec`/`opterr` variables that we needed to assign for our custom dataset before are no longer present. Instead, they are replaced by the `aplev`/`aperr` variables. Furthermore, since the wavelength is the same for every pixel in photometric data, the `wave_1d` variable should just be set to the effective wavelength of the filter for a single, arbitrary `x` pixel coordinate.\n", "\n", "So, let's fake another dataset, but this time for photometry:" ] }, { "cell_type": "code", "execution_count": 11, "id": "2a4bc420-b937-4b17-abde-2f15afed1a06", "metadata": {}, "outputs": [], "source": [ "# Assign your data to some variables\n", "time = np.linspace(59694.0, 59694.1, 100)\n", "time_units = 'MJD'\n", "aplev = np.ones(len(time))\n", "aperr = np.ones(len(time))*0.01\n", "wave_1d = np.array([2.1,])\n", "wave_units = 'microns'\n", "flux_units = 'e-'\n", "\n", "# Create an empty dataset\n", "photom_dataset = makeDataset()\n", "\n", "# Assign the variables to the dataset\n", "photom_dataset['time'] = time\n", "photom_dataset['time'].attrs['time_units'] = time_units\n", "photom_dataset['x'] = [0,]\n", "photom_dataset['aplev'] = (['time'], aplev)\n", "photom_dataset['aplev'].attrs['flux_units'] = flux_units\n", "photom_dataset['aplev'].attrs['time_units'] = time_units\n", "photom_dataset['aperr'] = (['time'], aperr)\n", "photom_dataset['aperr'].attrs['flux_units'] = flux_units\n", "photom_dataset['aperr'].attrs['time_units'] = time_units\n", "photom_dataset['wave_1d'] = (['x'], wave_1d)\n", "photom_dataset['wave_1d'].attrs['wave_units'] = wave_units" ] }, { "cell_type": "markdown", "id": "86160e0e-4ffb-4a93-9354-c26043a0c1a6", "metadata": {}, "source": [ "Like before, we also need to assign some attributes to the dataset. These stay the same as for spectroscopic data, however, we also need to add an important attribute to indicate that this is a photometric dataset. Let's do that now, and take a look at everything. " ] }, { "cell_type": "code", "execution_count": 12, "id": "04206681-8b70-45f4-9b9f-f06293ff7876", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Dimensions: (time: 100, x: 1)\n", "Coordinates:\n", " * time (time) float64 5.969e+04 5.969e+04 ... 5.969e+04 5.969e+04\n", " * x (x) int64 0\n", "Data variables:\n", " aplev (time) float64 1.0 1.0 1.0 1.0 1.0 1.0 ... 1.0 1.0 1.0 1.0 1.0 1.0\n", " aperr (time) float64 0.01 0.01 0.01 0.01 0.01 ... 0.01 0.01 0.01 0.01\n", " wave_1d (x) float64 2.1\n", "Attributes:\n", " inst: nircam\n", " data_format: custom\n", " photometry: True\n" ] } ], "source": [ "photom_dataset.attrs['inst'] = 'nircam'\n", "photom_dataset.attrs['data_format'] = 'custom'\n", "photom_dataset.attrs['photometry'] = True\n", "\n", "print(photom_dataset)" ] }, { "cell_type": "markdown", "id": "64641a15-954f-495f-a002-e956b1354b8a", "metadata": {}, "source": [ "
\n", "Great! Finally, we can save the file just like before. " ] }, { "cell_type": "code", "execution_count": 13, "id": "b3d2293a-0ae6-4c9d-b8a6-17505d6d108a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Finished writing to /Users/acarter/Documents/TRANSITS/EUREKA/ALTERNATE_INPUTS/NIRCam_TESTING/Custom_Photom_Input/S3_photom_dataset_SpecData.h5\n" ] }, { "data": { "text/plain": [ "True" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Make sure we have made the directory to save the file to\n", "save_dir = base_dir+'Custom_Photom_Input/'\n", "if not os.path.exists(save_dir):\n", " os.makedirs(save_dir)\n", "save_file = save_dir+'S3_photom_dataset_SpecData.h5' \n", "\n", "# Save the file\n", "writeXR(save_file, photom_dataset, verbose=True)" ] } ], "metadata": { "kernelspec": { "display_name": "Python [conda env:eureka] *", "language": "python", "name": "conda-env-eureka-py" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.7" } }, "nbformat": 4, "nbformat_minor": 5 }