Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Jupyter Notebook for CARRA on Height Levels - Downloading datasets and Preparing them for analysis #180

Open
FabioMangini opened this issue Jan 28, 2025 · 8 comments
Labels

Comments

@FabioMangini
Copy link

FabioMangini commented Jan 28, 2025

Notebook description

Good morning,

I would like to create a Jupyter Notebook that evaluates the ability of ERA5 and CARRA to estimate the energy production of onshore wind farms in northern Norway. I am writing to you because I was wondering if you could help me with downloading the datasets and preparing them for the analysis.

The energy produced by a wind turbine is related to the wind speed at the turbine's height. So, I was wondering whether you could create a Jupyter Notebook that downloads the:

  1. zonal and meridional components of the wind field from the ERA5 hourly reanalysis data on single levels at 100m height (https://cds.climate.copernicus.eu/datasets/reanalysis-era5-single-levels?tab=download)

  2. wind speed from CARRA-West and CARRA-East on height levels at 75m and 100m height (https://cds.climate.copernicus.eu/datasets/reanalysis-carra-height-levels?tab=download)

since January 1992.

Once the datasets have been downloaded, I was wondering whether you compute the climatological mean of the wind speed from CARRA-West and CARRA-East? They would help identify the locations that are most suitable for setting up new wind farms by showing climatological wind speed values.

The Jupyter Notebook would also like to focus on individual wind farms and understand how well ERA5 and CARRA can estimate their energy production. For simplicity, the Jupyter Notebook would first consider a single wind farm. This is the Dønnesfjord wind farm, which is located at 70.63°N and 22.46°E and became operative in 2022. So, I was wondering if you could create a script that extracts the wind speed from ERA5 and CARRA-East at 70.63°N and 22.46°E since January 2022.

To conclude, I was also wondering if it were possible for you to use python to download and store into a pandas DataFrame the wind power produced by the Norwegian wind farm. The data are available at: https://www.nve.no/media/16891/vindprod2002-2023_kraftverk_utcplus1.xlsx.

Thank you very much.

Notebook link or upload

wind_power_estimation.ipynb.zip

Anything else we need to know?

No response

Environment

name: wp5
channels:

  • conda-forge
    dependencies:
  • _libgcc_mutex=0.1=conda_forge
  • _openmp_mutex=4.5=2_gnu
  • accessible-pygments=0.0.5=pyhd8ed1ab_1
  • affine=2.4.0=pyhd8ed1ab_1
  • aiohttp=3.9.5=py311h459d7ec_0
  • aiosignal=1.3.2=pyhd8ed1ab_0
  • alabaster=0.7.16=pyhd8ed1ab_0
  • alsa-lib=1.2.13=hb9d3cd8_0
  • annotated-types=0.7.0=pyhd8ed1ab_1
  • ansicolors=1.1.8=pyhd8ed1ab_0
  • anyio=4.8.0=pyhd8ed1ab_0
  • aom=3.9.1=hac33072_0
  • argon2-cffi=23.1.0=pyhd8ed1ab_1
  • argon2-cffi-bindings=21.2.0=py311h9ecbd09_5
  • array-api-compat=1.10.0=pyhd8ed1ab_0
  • arrow=1.3.0=pyhd8ed1ab_1
  • asttokens=3.0.0=pyhd8ed1ab_1
  • async-lru=2.0.4=pyhd8ed1ab_1
  • attrs=24.3.0=pyh71513ae_0
  • aws-c-auth=0.8.1=h205f482_0
  • aws-c-cal=0.8.1=h1a47875_3
  • aws-c-common=0.10.6=hb9d3cd8_0
  • aws-c-compression=0.3.0=h4e1184b_5
  • aws-c-event-stream=0.5.0=h7959bf6_11
  • aws-c-http=0.9.2=hefd7a92_4
  • aws-c-io=0.15.3=h173a860_6
  • aws-c-mqtt=0.11.0=h11f4f37_12
  • aws-c-s3=0.7.9=he1b24dc_1
  • aws-c-sdkutils=0.2.2=h4e1184b_0
  • aws-checksums=0.2.2=h4e1184b_4
  • aws-crt-cpp=0.29.9=he0e7f3f_2
  • aws-sdk-cpp=1.11.489=h4d475cb_0
  • azure-core-cpp=1.14.0=h5cfcd09_0
  • azure-identity-cpp=1.10.0=h113e628_0
  • azure-storage-blobs-cpp=12.13.0=h3cf044e_1
  • azure-storage-common-cpp=12.8.0=h736e048_1
  • azure-storage-files-datalake-cpp=12.12.0=ha633028_1
  • babel=2.16.0=pyhd8ed1ab_1
  • beautifulsoup4=4.12.3=pyha770c72_1
  • bleach=6.2.0=pyh29332c3_4
  • bleach-with-css=6.2.0=h82add2a_4
  • blosc=1.21.6=he440d0b_1
  • bokeh=3.6.2=pyhd8ed1ab_1
  • bottleneck=1.4.2=py311h9f3472d_0
  • branca=0.8.1=pyhd8ed1ab_0
  • brotli=1.1.0=hb9d3cd8_2
  • brotli-bin=1.1.0=hb9d3cd8_2
  • brotli-python=1.1.0=py311hfdbb021_2
  • bzip2=1.0.8=h4bc722e_7
  • c-ares=1.34.4=hb9d3cd8_0
  • ca-certificates=2024.12.14=hbcca054_0
  • cached-property=1.5.2=hd8ed1ab_1
  • cached_property=1.5.2=pyha770c72_1
  • cairo=1.18.2=h3394656_1
  • cartopy=0.24.0=py311h7db5c69_0
  • cdsapi=0.7.5=pyhd8ed1ab_1
  • certifi=2024.12.14=pyhd8ed1ab_0
  • cf-units=3.3.0=py311h9f3472d_0
  • cf_xarray=0.10.0=pyhd8ed1ab_2
  • cffi=1.17.1=py311hf29c0ef_0
  • cfgrib=0.9.15.0=pyhd8ed1ab_0
  • cftime=1.6.4=py311h9f3472d_1
  • charset-normalizer=3.4.1=pyhd8ed1ab_0
  • click=8.1.8=pyh707e725_0
  • click-plugins=1.1.1=pyhd8ed1ab_1
  • cligj=0.7.2=pyhd8ed1ab_2
  • cloudpickle=3.1.1=pyhd8ed1ab_0
  • cmocean=4.0.3=pyhd8ed1ab_1
  • colorama=0.4.6=pyhd8ed1ab_1
  • colorspacious=1.1.2=pyhecae5ae_1
  • comm=0.2.2=pyhd8ed1ab_1
  • contourpy=1.3.1=py311hd18a35c_0
  • crc32c=2.7.1=py311h9ecbd09_0
  • cycler=0.12.1=pyhd8ed1ab_1
  • cyrus-sasl=2.1.27=h54b06d7_7
  • cytoolz=1.0.1=py311h9ecbd09_0
  • dask=2025.1.0=pyhd8ed1ab_0
  • dask-core=2025.1.0=pyhd8ed1ab_0
  • datapi=0.1.2=pyhd8ed1ab_0
  • dav1d=1.2.1=hd590300_0
  • dbus=1.13.6=h5008d03_3
  • debugpy=1.8.12=py311hfdbb021_0
  • decorator=5.1.1=pyhd8ed1ab_1
  • defusedxml=0.7.1=pyhd8ed1ab_0
  • deprecated=1.2.15=pyhd8ed1ab_1
  • distributed=2025.1.0=pyhd8ed1ab_0
  • docutils=0.20.1=py311h38be061_3
  • donfig=0.8.1.post1=pyhd8ed1ab_1
  • double-conversion=3.3.0=h59595ed_0
  • earthkit-data=0.12.1=pyhd8ed1ab_0
  • earthkit-geo=0.3.0=pyhd8ed1ab_1
  • entrypoints=0.4=pyhd8ed1ab_1
  • esmf=8.7.0=nompi_h6063b07_1
  • esmpy=8.7.0=pyhecae5ae_1
  • exceptiongroup=1.2.2=pyhd8ed1ab_1
  • executing=2.1.0=pyhd8ed1ab_1
  • expat=2.6.4=h5888daf_0
  • filelock=3.17.0=pyhd8ed1ab_0
  • findlibs=0.0.5=pyhd8ed1ab_0
  • flox=0.9.15=pyhd8ed1ab_1
  • folium=0.19.4=pyhd8ed1ab_0
  • font-ttf-dejavu-sans-mono=2.37=hab24e00_0
  • font-ttf-inconsolata=3.000=h77eed37_0
  • font-ttf-source-code-pro=2.038=h77eed37_0
  • font-ttf-ubuntu=0.83=h77eed37_3
  • fontconfig=2.15.0=h7e30c49_1
  • fonts-conda-ecosystem=1=0
  • fonts-conda-forge=1=0
  • fonttools=4.55.6=py311h2dc5d0c_0
  • fqdn=1.5.1=pyhd8ed1ab_1
  • freeglut=3.2.2=ha6d2627_3
  • freetype=2.12.1=h267a509_2
  • freexl=2.0.0=h9dce30a_2
  • frozenlist=1.5.0=py311h2dc5d0c_1
  • fsspec=2024.12.0=pyhd8ed1ab_0
  • geopandas=1.0.1=pyhd8ed1ab_3
  • geopandas-base=1.0.1=pyha770c72_3
  • geos=3.13.0=h5888daf_0
  • geotiff=1.7.3=h77b800c_3
  • gflags=2.2.2=h5888daf_1005
  • giflib=5.2.2=hd590300_0
  • glog=0.7.1=hbabe93e_0
  • graphite2=1.3.13=h59595ed_1003
  • greenlet=3.1.1=py311hfdbb021_1
  • h11=0.14.0=pyhd8ed1ab_1
  • h2=4.1.0=pyhd8ed1ab_1
  • harfbuzz=10.2.0=h4bba637_0
  • hdf4=4.2.15=h2a13503_7
  • hdf5=1.14.4=nompi_h2d575fe_105
  • hpack=4.1.0=pyhd8ed1ab_0
  • httpcore=1.0.7=pyh29332c3_1
  • httpx=0.28.1=pyhd8ed1ab_0
  • hyperframe=6.1.0=pyhd8ed1ab_0
  • icu=75.1=he02047a_0
  • idna=3.10=pyhd8ed1ab_1
  • imagesize=1.4.1=pyhd8ed1ab_0
  • importlib-metadata=8.6.1=pyha770c72_0
  • importlib_resources=6.5.2=pyhd8ed1ab_0
  • ipykernel=6.29.5=pyh3099207_0
  • ipython=8.31.0=pyh707e725_0
  • isoduration=20.11.0=pyhd8ed1ab_1
  • jasper=4.2.4=h536e39c_0
  • jedi=0.19.2=pyhd8ed1ab_1
  • jinja2=3.1.5=pyhd8ed1ab_0
  • joblib=1.4.2=pyhd8ed1ab_1
  • json-c=0.18=h6688a6e_0
  • json5=0.10.0=pyhd8ed1ab_1
  • jsonpointer=3.0.0=py311h38be061_1
  • jsonschema=4.23.0=pyhd8ed1ab_1
  • jsonschema-specifications=2024.10.1=pyhd8ed1ab_1
  • jsonschema-with-format-nongpl=4.23.0=hd8ed1ab_1
  • jupyter-book=1.0.3=pyhd8ed1ab_1
  • jupyter-cache=1.0.1=pyhff2d567_0
  • jupyter-lsp=2.2.5=pyhd8ed1ab_1
  • jupyter-server-proxy=4.4.0=pyhd8ed1ab_1
  • jupyter_client=8.6.3=pyhd8ed1ab_1
  • jupyter_core=5.7.2=pyh31011fe_1
  • jupyter_events=0.11.0=pyhd8ed1ab_0
  • jupyter_server=2.15.0=pyhd8ed1ab_0
  • jupyter_server_terminals=0.5.3=pyhd8ed1ab_1
  • jupyterlab=4.3.4=pyhd8ed1ab_0
  • jupyterlab-myst=2.4.2=pyhd8ed1ab_1
  • jupyterlab_pygments=0.3.0=pyhd8ed1ab_2
  • jupyterlab_server=2.27.3=pyhd8ed1ab_1
  • keyutils=1.6.1=h166bdaf_0
  • kiwisolver=1.4.7=py311hd18a35c_0
  • krb5=1.21.3=h659f571_0
  • latexcodec=2.0.1=pyh9f0ad1d_0
  • lcms2=2.16=hb7c19ff_0
  • ld_impl_linux-64=2.43=h712a8e2_2
  • lerc=4.0.0=h27087fc_0
  • libabseil=20240722.0=cxx17_hbbce691_4
  • libaec=1.1.3=h59595ed_0
  • libarchive=3.7.7=h4585015_3
  • libarrow=19.0.0=h00a82cf_7_cpu
  • libarrow-acero=19.0.0=hcb10f89_7_cpu
  • libarrow-dataset=19.0.0=hcb10f89_7_cpu
  • libarrow-substrait=19.0.0=h08228c5_7_cpu
  • libavif16=1.1.1=h1909e37_2
  • libblas=3.9.0=26_linux64_openblas
  • libbrotlicommon=1.1.0=hb9d3cd8_2
  • libbrotlidec=1.1.0=hb9d3cd8_2
  • libbrotlienc=1.1.0=hb9d3cd8_2
  • libcblas=3.9.0=26_linux64_openblas
  • libclang-cpp19.1=19.1.7=default_hb5137d0_1
  • libclang13=19.1.7=default_h9c6a7e4_1
  • libcrc32c=1.1.2=h9c3ff4c_0
  • libcups=2.3.3=h4637d8d_4
  • libcurl=8.11.1=h332b0f4_0
  • libde265=1.0.15=h00ab1b0_0
  • libdeflate=1.23=h4ddbbb0_0
  • libdrm=2.4.124=hb9d3cd8_0
  • libedit=3.1.20240808=pl5321h7949ede_0
  • libegl=1.7.0=ha4b6fd6_2
  • libev=4.33=hd590300_2
  • libevent=2.1.12=hf998b51_1
  • libexpat=2.6.4=h5888daf_0
  • libffi=3.4.2=h7f98852_5
  • libgcc=14.2.0=h77fa898_1
  • libgcc-ng=14.2.0=h69a702a_1
  • libgdal-core=3.10.1=h3359108_2
  • libgfortran=14.2.0=h69a702a_1
  • libgfortran5=14.2.0=hd5240d6_1
  • libgl=1.7.0=ha4b6fd6_2
  • libglib=2.82.2=h2ff4ddf_1
  • libglu=9.0.3=h03adeef_0
  • libglvnd=1.7.0=ha4b6fd6_2
  • libglx=1.7.0=ha4b6fd6_2
  • libgomp=14.2.0=h77fa898_1
  • libgoogle-cloud=2.34.0=h2b5623c_0
  • libgoogle-cloud-storage=2.34.0=h0121fbd_0
  • libgrpc=1.67.1=h25350d4_1
  • libheif=1.19.5=gpl_hc21c24c_100
  • libiconv=1.17=hd590300_2
  • libjpeg-turbo=3.0.0=hd590300_1
  • libkml=1.3.0=hf539b9f_1021
  • liblapack=3.9.0=26_linux64_openblas
  • libllvm14=14.0.6=hcd5def8_4
  • libllvm19=19.1.7=ha7bfdaf_1
  • liblzma=5.6.3=hb9d3cd8_1
  • libnetcdf=4.9.2=nompi_h5ddbaa4_116
  • libnghttp2=1.64.0=h161d5f1_0
  • libnsl=2.0.1=hd590300_0
  • libntlm=1.8=hb9d3cd8_0
  • libopenblas=0.3.28=pthreads_h94d23a6_1
  • libopengl=1.7.0=ha4b6fd6_2
  • libopentelemetry-cpp=1.18.0=hfcad708_1
  • libopentelemetry-cpp-headers=1.18.0=ha770c72_1
  • libparquet=19.0.0=h081d1f1_7_cpu
  • libpciaccess=0.18=hd590300_0
  • libpng=1.6.46=h943b412_0
  • libpq=17.2=h3b95a9b_1
  • libprotobuf=5.28.3=h6128344_1
  • libre2-11=2024.07.02=hbbce691_2
  • librttopo=1.1.0=h97f6797_17
  • libsodium=1.0.20=h4ab18f5_0
  • libspatialite=5.1.0=h1b4f908_12
  • libsqlite=3.48.0=hee588c1_1
  • libssh2=1.11.1=hf672d98_0
  • libstdcxx=14.2.0=hc0a3c3a_1
  • libstdcxx-ng=14.2.0=h4852527_1
  • libthrift=0.21.0=h0e7cc3e_0
  • libtiff=4.7.0=hd9ff511_3
  • libudunits2=2.2.28=h40f5838_3
  • libutf8proc=2.10.0=h4c51ac1_0
  • libuuid=2.38.1=h0b41bf4_0
  • libwebp-base=1.5.0=h851e524_0
  • libxcb=1.17.0=h8a09558_0
  • libxcrypt=4.4.36=hd590300_1
  • libxkbcommon=1.7.0=h2c5496b_1
  • libxml2=2.13.5=h8d12d68_1
  • libxslt=1.1.39=h76b75d6_0
  • libzip=1.11.2=h6991a6a_0
  • libzlib=1.3.1=hb9d3cd8_2
  • linkify-it-py=2.0.3=pyhd8ed1ab_1
  • llvmlite=0.43.0=py311h9c9ff8c_1
  • locket=1.0.0=pyhd8ed1ab_0
  • lru-dict=1.3.0=py311h9ecbd09_1
  • lz4=4.3.3=py311h8c6ae76_2
  • lz4-c=1.10.0=h5888daf_1
  • lzo=2.10=hd590300_1001
  • mapclassify=2.8.1=pyhd8ed1ab_1
  • markdown=3.6=pyhd8ed1ab_0
  • markdown-it-py=3.0.0=pyhd8ed1ab_1
  • markupsafe=3.0.2=py311h2dc5d0c_1
  • matplotlib=3.10.0=py311h38be061_0
  • matplotlib-base=3.10.0=py311h2b939e6_0
  • matplotlib-inline=0.1.7=pyhd8ed1ab_1
  • mdit-py-plugins=0.4.2=pyhd8ed1ab_1
  • mdurl=0.1.2=pyhd8ed1ab_1
  • minizip=4.0.7=h05a5f5f_3
  • mistune=3.1.0=pyhd8ed1ab_0
  • msgpack-python=1.1.0=py311hd18a35c_0
  • multidict=6.1.0=py311h2dc5d0c_2
  • multiurl=0.3.3=pyhd8ed1ab_1
  • munkres=1.1.4=pyh9f0ad1d_0
  • mysql-common=9.0.1=h266115a_4
  • mysql-libs=9.0.1=he0572af_4
  • myst-nb=1.1.2=pyhd8ed1ab_1
  • myst-parser=2.0.0=pyhd8ed1ab_0
  • nbclient=0.10.2=pyhd8ed1ab_0
  • nbconvert=7.16.5=hd8ed1ab_1
  • nbconvert-core=7.16.5=pyhd8ed1ab_1
  • nbconvert-pandoc=7.16.5=hd8ed1ab_1
  • nbformat=5.10.4=pyhd8ed1ab_1
  • nc-time-axis=1.4.1=pyhd8ed1ab_1
  • ncurses=6.5=h2d0b736_2
  • nest-asyncio=1.6.0=pyhd8ed1ab_1
  • netcdf-fortran=4.6.1=nompi_ha5d1325_108
  • netcdf4=1.7.2=nompi_py311h7c29e4f_101
  • networkx=3.4.2=pyh267e887_2
  • nlohmann_json=3.11.3=he02047a_1
  • notebook-shim=0.2.4=pyhd8ed1ab_1
  • numba=0.60.0=py311h4bc866e_0
  • numcodecs=0.15.0=py311h7db5c69_0
  • numpy=2.0.2=py311h71ddf71_1
  • numpy_groupies=0.11.2=pyhd8ed1ab_1
  • openjpeg=2.5.3=h5fbd93e_0
  • openldap=2.6.9=he970967_0
  • openssl=3.4.0=h7b32b05_1
  • orc=2.0.3=h12ee42a_2
  • overrides=7.7.0=pyhd8ed1ab_1
  • packaging=24.2=pyhd8ed1ab_2
  • pandas=2.2.3=py311h7db5c69_1
  • pandoc=3.6.2=ha770c72_0
  • pandocfilters=1.5.0=pyhd8ed1ab_0
  • papermill=2.6.0=pyhd8ed1ab_1
  • parso=0.8.4=pyhd8ed1ab_1
  • partd=1.4.2=pyhd8ed1ab_0
  • patsy=1.0.1=pyhd8ed1ab_1
  • pcre2=10.44=hba22ea6_2
  • pexpect=4.9.0=pyhd8ed1ab_1
  • pickleshare=0.7.5=pyhd8ed1ab_1004
  • pillow=11.1.0=py311h1322bbf_0
  • pip=24.3.1=pyh8b19718_2
  • pixman=0.44.2=h29eaf8c_0
  • pkgutil-resolve-name=1.3.10=pyhd8ed1ab_2
  • platformdirs=4.3.6=pyhd8ed1ab_1
  • plotly=5.24.1=pyhd8ed1ab_1
  • pooch=1.8.2=pyhd8ed1ab_1
  • proj=9.5.1=h0054346_0
  • prometheus-cpp=1.3.0=ha5d0236_0
  • prometheus_client=0.21.1=pyhd8ed1ab_0
  • prompt-toolkit=3.0.50=pyha770c72_0
  • propcache=0.2.1=py311h2dc5d0c_1
  • properscoring=0.1=pyhd8ed1ab_1
  • psutil=6.1.1=py311h9ecbd09_0
  • pthread-stubs=0.4=hb9d3cd8_1002
  • ptyprocess=0.7.0=pyhd8ed1ab_1
  • pure_eval=0.2.3=pyhd8ed1ab_1
  • pwlf=2.4.0=py311h38be061_0
  • pyarrow=19.0.0=py311h38be061_0
  • pyarrow-core=19.0.0=py311h4854187_0_cpu
  • pybtex=0.24.0=pyhd8ed1ab_3
  • pybtex-docutils=1.0.3=py311h38be061_2
  • pycparser=2.22=pyh29332c3_1
  • pydantic=2.10.6=pyh3cfb1c2_0
  • pydantic-core=2.27.2=py311h9e33e62_0
  • pydantic-settings=2.7.1=pyh3cfb1c2_0
  • pydata-sphinx-theme=0.16.1=pyhd8ed1ab_0
  • pydoe=0.3.8=py_1
  • pygments=2.19.1=pyhd8ed1ab_0
  • pyogrio=0.10.0=py311hf6089d3_1
  • pyparsing=3.2.1=pyhd8ed1ab_0
  • pyproj=3.7.0=py311h0f98d5a_0
  • pyshp=2.3.1=pyhd8ed1ab_1
  • pyside6=6.8.1=py311h9053184_0
  • pysocks=1.7.1=pyha55dd90_7
  • python=3.11.11=h9e4cc4f_1_cpython
  • python-dateutil=2.9.0.post0=pyhff2d567_1
  • python-dotenv=1.0.1=pyhd8ed1ab_1
  • python-fastjsonschema=2.21.1=pyhd8ed1ab_0
  • python-json-logger=2.0.7=pyhd8ed1ab_0
  • python-tzdata=2025.1=pyhd8ed1ab_0
  • python_abi=3.11=5_cp311
  • pytz=2024.1=pyhd8ed1ab_0
  • pyyaml=6.0.2=py311h2dc5d0c_2
  • pyzmq=26.2.0=py311h7deb3e3_3
  • qhull=2020.2=h434a139_5
  • qt6-main=6.8.1=h588cce1_2
  • rasterio=1.4.3=py311h5394301_0
  • rav1e=0.6.6=he8a937b_2
  • re2=2024.07.02=h9925aae_2
  • readline=8.2=h8228510_1
  • referencing=0.36.1=pyhd8ed1ab_0
  • regionmask=0.13.0=pyhd8ed1ab_0
  • requests=2.32.3=pyhd8ed1ab_1
  • rfc3339-validator=0.1.4=pyhd8ed1ab_1
  • rfc3986-validator=0.1.1=pyh9f0ad1d_0
  • rich=13.9.4=pyhd8ed1ab_1
  • rioxarray=0.18.2=pyhd8ed1ab_0
  • rpds-py=0.22.3=py311h9e33e62_0
  • s2n=1.5.11=h072c03f_0
  • scikit-learn=1.6.1=py311h57cc02b_0
  • scipy=1.15.1=py311hc1ac118_0
  • seaborn=0.13.2=hd8ed1ab_3
  • seaborn-base=0.13.2=pyhd8ed1ab_3
  • send2trash=1.8.3=pyh0d859eb_1
  • setuptools=75.8.0=pyhff2d567_0
  • shapely=2.0.6=py311h2fdb869_2
  • shellingham=1.5.4=pyhd8ed1ab_1
  • simpervisor=1.0.0=pyhd8ed1ab_1
  • six=1.17.0=pyhd8ed1ab_0
  • snappy=1.2.1=h8bd8927_1
  • sniffio=1.3.1=pyhd8ed1ab_1
  • snowballstemmer=2.2.0=pyhd8ed1ab_0
  • snuggs=1.4.7=pyhd8ed1ab_2
  • sortedcontainers=2.4.0=pyhd8ed1ab_0
  • soupsieve=2.5=pyhd8ed1ab_1
  • sparse=0.15.5=pyh72ffeb9_0
  • sphinx=7.4.7=pyhd8ed1ab_0
  • sphinx-book-theme=1.1.3=pyhd8ed1ab_1
  • sphinx-comments=0.0.3=pyhd8ed1ab_1
  • sphinx-copybutton=0.5.2=pyhd8ed1ab_1
  • sphinx-design=0.6.1=pyhd8ed1ab_2
  • sphinx-external-toc=1.0.1=pyhd8ed1ab_1
  • sphinx-jupyterbook-latex=1.0.0=pyhd8ed1ab_1
  • sphinx-multitoc-numbering=0.1.3=pyhd8ed1ab_1
  • sphinx-thebe=0.3.1=pyhd8ed1ab_1
  • sphinx-togglebutton=0.3.2=pyhd8ed1ab_0
  • sphinxcontrib-applehelp=2.0.0=pyhd8ed1ab_1
  • sphinxcontrib-bibtex=2.6.3=pyhd8ed1ab_1
  • sphinxcontrib-devhelp=2.0.0=pyhd8ed1ab_1
  • sphinxcontrib-htmlhelp=2.1.0=pyhd8ed1ab_1
  • sphinxcontrib-jsmath=1.0.1=pyhd8ed1ab_1
  • sphinxcontrib-qthelp=2.0.0=pyhd8ed1ab_1
  • sphinxcontrib-serializinghtml=1.1.10=pyhd8ed1ab_1
  • sqlalchemy=2.0.37=py311h9ecbd09_0
  • sqlite=3.48.0=h9eae976_1
  • stack_data=0.6.3=pyhd8ed1ab_1
  • statsmodels=0.14.4=py311h9f3472d_0
  • structlog=24.4.0=pyhd8ed1ab_1
  • svt-av1=2.3.0=h5888daf_0
  • tabulate=0.9.0=pyhd8ed1ab_2
  • tblib=3.0.0=pyhd8ed1ab_1
  • tenacity=9.0.0=pyhd8ed1ab_1
  • terminado=0.18.1=pyh0d859eb_0
  • threadpoolctl=3.5.0=pyhc1e730c_0
  • tinycss2=1.4.0=pyhd8ed1ab_0
  • tk=8.6.13=noxft_h4845f30_101
  • toml=0.10.2=pyhd8ed1ab_1
  • tomli=2.2.1=pyhd8ed1ab_1
  • toolz=1.0.0=pyhd8ed1ab_1
  • tornado=6.4.2=py311h9ecbd09_0
  • tqdm=4.67.1=pyhd8ed1ab_1
  • traitlets=5.14.3=pyhd8ed1ab_1
  • typer=0.15.1=pyhd8ed1ab_0
  • typer-slim=0.15.1=pyhd8ed1ab_0
  • typer-slim-standard=0.15.1=hd8ed1ab_0
  • types-python-dateutil=2.9.0.20241206=pyhd8ed1ab_0
  • typing-extensions=4.12.2=hd8ed1ab_1
  • typing_extensions=4.12.2=pyha770c72_1
  • typing_utils=0.1.0=pyhd8ed1ab_1
  • tzdata=2025a=h78e105d_0
  • uc-micro-py=1.0.3=pyhd8ed1ab_1
  • udunits2=2.2.28=h40f5838_3
  • unicodedata2=16.0.0=py311h9ecbd09_0
  • uri-template=1.3.0=pyhd8ed1ab_1
  • uriparser=0.9.8=hac33072_0
  • urllib3=2.3.0=pyhd8ed1ab_0
  • wayland=1.23.1=h3e06ad9_0
  • wcwidth=0.2.13=pyhd8ed1ab_1
  • webcolors=24.11.1=pyhd8ed1ab_0
  • webencodings=0.5.1=pyhd8ed1ab_3
  • websocket-client=1.8.0=pyhd8ed1ab_1
  • wheel=0.45.1=pyhd8ed1ab_1
  • wrapt=1.17.2=py311h9ecbd09_0
  • x265=3.5=h924138e_3
  • xarray=2025.1.1=pyhd8ed1ab_0
  • xarraymannkendall=1.4.5=pyhd8ed1ab_0
  • xcb-util=0.4.1=hb711507_2
  • xcb-util-cursor=0.1.5=hb9d3cd8_0
  • xcb-util-image=0.4.0=hb711507_2
  • xcb-util-keysyms=0.4.1=hb711507_0
  • xcb-util-renderutil=0.3.10=hb711507_0
  • xcb-util-wm=0.4.2=hb711507_0
  • xerces-c=3.2.5=h988505b_2
  • xesmf=0.8.8=pyhd8ed1ab_1
  • xhistogram=0.3.2=pyhd8ed1ab_0
  • xkeyboard-config=2.43=hb9d3cd8_0
  • xorg-libice=1.1.2=hb9d3cd8_0
  • xorg-libsm=1.2.5=he73a12e_0
  • xorg-libx11=1.8.10=h4f16b4b_1
  • xorg-libxau=1.0.12=hb9d3cd8_0
  • xorg-libxcomposite=0.4.6=hb9d3cd8_2
  • xorg-libxcursor=1.2.3=hb9d3cd8_0
  • xorg-libxdamage=1.1.6=hb9d3cd8_0
  • xorg-libxdmcp=1.1.5=hb9d3cd8_0
  • xorg-libxext=1.3.6=hb9d3cd8_0
  • xorg-libxfixes=6.0.1=hb9d3cd8_0
  • xorg-libxi=1.8.2=hb9d3cd8_0
  • xorg-libxrandr=1.5.4=hb9d3cd8_0
  • xorg-libxrender=0.9.12=hb9d3cd8_0
  • xorg-libxtst=1.2.5=hb9d3cd8_3
  • xorg-libxxf86vm=1.1.6=hb9d3cd8_0
  • xskillscore=0.0.26=pyhd8ed1ab_1
  • xyzservices=2025.1.0=pyhd8ed1ab_0
  • yaml=0.2.5=h7f98852_2
  • yarl=1.18.3=py311h2dc5d0c_1
  • zarr=3.0.1=pyhd8ed1ab_0
  • zeromq=4.3.5=h3b0a872_7
  • zict=3.0.0=pyhd8ed1ab_1
  • zipp=3.21.0=pyhd8ed1ab_1
  • zlib=1.3.1=hb9d3cd8_2
  • zstandard=0.23.0=py311hbc35293_1
  • zstd=1.5.6=ha6fb4c9_0
  • pip:
    • alembic==1.14.1
    • c3s-eqc-automatic-quality-control==0.1.2.dev114+g8ac355a
    • cacholote==1.5.0
    • cgul==0.0.4
    • earthkit-meteo==0.3.0
    • eccodes==2.39.1
    • ecmwflibs==0.6.3
    • kaleido==0.2.1
    • mako==1.3.8
    • pdbufr==0.12.2
    • pymannkendall==1.4.3
    • sqlalchemy-utils==0.41.2
      prefix: /data/common/miniforge3/envs/wp5
@malmans2
Copy link
Member

Hi @FabioMangini,

The notebook you shared is empty.

@FabioMangini
Copy link
Author

FabioMangini commented Jan 29, 2025

Hi @malmans2,

Here's a new notebook:

I created it to download:

  1. zonal and meridional components of the wind field from the ERA5 hourly reanalysis data on single levels at 100m height (https://cds.climate.copernicus.eu/datasets/reanalysis-era5-single-levels?tab=download)

  2. wind speed from CARRA-West and CARRA-East on height levels at 75m and 100m height (https://cds.climate.copernicus.eu/datasets/reanalysis-carra-height-levels?tab=download)

between September 1990 (in the text above I mistakenly referred to January 1992) and November 2024 .

I have an issue with this script. I get the following the error message when I run it:

  • Exception: Missing/incomplete configuration file: /data/wp5/.cdsapirc

I was wondering whether you could help me solve this issue.

@malmans2
Copy link
Member

Hi @FabioMangini,

Is the notebook intended to download all the data you need? If you'd like me to cache the data, I can do that and will notify you once it's available on the VM.

Regarding the issue:

In the old VM there used to be a shared .cdsapirc (I think it was Vincenzo's account), but there isn't anymore.
However, it's much better if you use your own .cdsapirc

Create your own config file in ~/mangini_fabio/.cdsapirc.
Then, add these lines in your notebooks when you need to download data:

import os

os.environ["CDSAPI_RC"] = os.path.expanduser("~/mangini_fabio/.cdsapirc")

If you don't know how to create your .cdsapirc, you can follow these instructions: https://github.com/ecmwf/cdsapi?tab=readme-ov-file#configure

@FabioMangini
Copy link
Author

Hi @malmans2,

Thanks for the explanation. I will follow the procedure you suggest.

Is the notebook intended to download all the data you need? If you'd like me to cache the data, I can do that and will notify you once it's available on the VM.

Yes, it would be great if you cached the data. Thanks!

I would like to ask you two more questions about the Jupyter Notebook:

  1. I need to download wind speed data from CARRA-East and CARRA-West at 75m and 100m height. They would help me to show time-averaged wind speed at each grid point of both CARRA's domains. How would you perform the time averaging without the kernel dying?

  2. I need the time series of wind speed from CARRA at the grid point closest to 70.63°N, 22.46°E. However, I'm concerned that extracting a time series from a dataset containing wind speed values for the entire CARRA domain at a 3-hour temporal resolution might cause the kernel to crash. Do you think this could be an issue? If so, how would you address it?

@malmans2
Copy link
Member

I need to download wind speed data from CARRA-East and CARRA-West at 75m and 100m height. They would help me to show time-averaged wind speed at each grid point of both CARRA's domains. How would you perform the time averaging without the kernel dying?

This can be tricky as we need to access the whole timeseries at once (i.e., we have to use download_and_transform(..., transform_chunks=False)). But I can't be sure until we try. One option is to implement an intermediate step that rechunks the data.

I need the time series of wind speed from CARRA at the grid point closest to 70.63°N, 22.46°E. However, I'm concerned that extracting a time series from a dataset containing wind speed values for the entire CARRA domain at a 3-hour temporal resolution might cause the kernel to crash. Do you think this could be an issue? If so, how would you address it?

This shouldn't be a problem as the data is chunked over time. Using download_and_transform(..., transform_chunks=True) should work fine.

Let's resume the discussion once the data is on disk. I have a few meetings now. I'll have a look to the code you shared in the afternoon and I will start downloading data if I have no questions.

@FabioMangini
Copy link
Author

Hi @malmans2,

perfect! Thank you very much.

@malmans2
Copy link
Member

Hi @FabioMangini,

I still didn't get to cache your data. I'm doing the same for other evaluators and it's taking longer than what I thought.
Hopefully I'll be able to start tomorrow, so we can let it go during the weekend if it's going to take a long time.

@FabioMangini
Copy link
Author

Hi @malmans2,

It is fine with me. Thank you very much for letting me know and for your help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants