BEGIN:VCALENDAR VERSION:2.0 PRODID:-//pretalx//talks.staging.osgeo.org//foss4g-2024//speaker//BZLYD8 BEGIN:VTIMEZONE TZID:-03 BEGIN:STANDARD DTSTART:20000101T000000 RRULE:FREQ=YEARLY;BYMONTH=1 TZNAME:-03 TZOFFSETFROM:-0300 TZOFFSETTO:-0300 END:STANDARD END:VTIMEZONE BEGIN:VEVENT UID:pretalx-foss4g-2024-MWYXS7@talks.staging.osgeo.org DTSTART;TZID=-03:20241205T160000 DTEND;TZID=-03:20241205T163000 DESCRIPTION:The IS_Agro project is an initiative focused on the critical ev aluation and subsequent adaptation of methodologies designed in global for ums\, with a view to their application in the national context based on th e development of new agro-socio-environmental metrics and indicators (IASs ) that aim to provide a more accurate and authentic representation of the agricultural landscape in the national territory. IASs are measures used t o monitor and evaluate agricultural performance related to social\, econom ic and environmental aspects\, thus having great importance in guiding mor e sustainable political strategies and agricultural practices\, whether by the public or private entity\, serving “to evaluate the performance of agriculture in terms of its environmental\, social and economic performanc e\, providing comparative data and information between federative entities or countries\, among several other applications” (EMBRAPA SOLOS\, 2023) . In this project\, IASs are developed by different teams specialized in t he proposed themes\, whose works are previously approved and published in the scientific arena. To automate data collection\, allocation\, calculati ons and constant updates of the IASs\, there is a team called the Digital Module\, which develops solutions for each indicator\, transforming them i nto digital algorithms. Structured\, semi-structured and unstructured regi stration data are collected and stored in a data lakehouse\, requiring a g reat deal of organization within the repository so that the data is always available and easily accessible. It was decided to implement the medallio n architecture (medal architecture)\, which consists of allocating data in three layers with different purposes\, while an open source platform was used for pipeline management and automation.\n\nThe conception of this pro ject as a digital platform linked to the Brazilian Agricultural Observator y aims to publish indicators and parameters derived from well-founded tech nical and scientific data\, capable of evaluating the effective performanc e of the national agricultural sector at the municipal or state level\, co ntributing to sectoral policies and planning and management processes aime d at building sustainable agriculture and the correct positioning of the c ountry on the international scene. Thus\, the general objective is to deve lop an intelligent environment that automates and manages the IAS pipeline s in a data storage organization environment based on the medallion archit ecture to be the basis of the data panel for publishing the indicators.\n\ nA data pipeline is a succession of connected phases that enable the colle ction\, storage\, modification\, analysis\, and representation of data\, w ith the purpose of acquiring meaningful insights and supporting informed c hoices (CALANCA\, 2023). A data lakehouse\, the destination of the project pipelines\, is “like a modern data platform built from a combination of a data lake and a data warehouse” (ORACLE CLOUD INFRASTRUCTURE\, 2023)\ , using “the flexible storage of unstructured data from a data lake and the management capabilities and tools of data warehouses\, and then strate gically deploying them together as a larger system” (ORACLE CLOUD INFRAS TRUCTURE\, 2023). The medallion architecture is the sequential structuring of data storage that aims to logically organize the data in the lakehouse \, aiming to incrementally and progressively improve the structure and qua lity of the data as it flows through the three layers of the architecture (ARQUITETURA medallion\, 2024). The terms bronze (raw data from the source )\, silver (transformation and validation of the data)\, and gold (refined and enriched data for use in projects) describe the quality of the data d uring the process (SKAYA et al\, 2024) . Pipeline management is performed by Apache Airflow (version 2.44)\, an open-source platform for developing\ , scheduling\, and monitoring batch-oriented workflows based on the Python programming language\, which allows you to create workflows connected to virtually any technology (WHAT is Airflow™?\, 2023). The Airflow executi on environment was structured in Docker\, an open-source platform that all ows you to create and manage containers as modular virtual machines that c ontain the essentials for their execution. The developed image is availabl e on GitHub.\nTo be confirmed\, the routines will be executed once a month . Raw data is collected by downloading and maintaining its original format \, with a hash of each file being recorded to indicate that the data has b een updated and download it again in the event of a change. This data is c leaned and processed as needed. At the end of the silver phase\, a tabular structure will be created with geocode (integer\, IBGE code of municipali ties or states)\, date (timestamp\, ISO 8601)\, source (text) and value (f loating point\, real number) and will be saved in the data lakehouse as .p arquet\, an open-source columnar storage format designed for highly compre ssed storage and efficient data retrieval\, providing improved performance for handling complex mass data (OVERVIEW\, 2022). The .parquet files save d in the data lake are available for use in the gold tier with one-to-many cardinality. In this last phase of the architecture\, the necessary calcu lations are performed for each source of the indicators\, with some source s that do not require calculations. The final phase is with the export of the gold data to tables in a project database in PostgreSQL\, being ready for use by an API developed internally that allows the provision of data f or the data panel to be developed (by another team) and published to socie ty from the project website.\n\nThis model has been adjusted and corrected throughout the development of the project in the Digital Module. Flexible \, it is now considered ready to receive any indicator developed by other teams\, as well as the development of the data panel for publication for u se by society. DTSTAMP:20260519T170558Z LOCATION:Room I SUMMARY:The Digital Module of the IS_Agro Project: Using the medallion arch itecture as a basis for automating pipeline execution routines in Apache A irflow - Carlos Eduardo Mota URL:https://talks.staging.osgeo.org/foss4g-2024/talk/MWYXS7/ END:VEVENT BEGIN:VEVENT UID:pretalx-foss4g-2024-BCYFPL@talks.staging.osgeo.org DTSTART;TZID=-03:20241206T104500 DTEND;TZID=-03:20241206T111500 DESCRIPTION:This talk presents case studies of deploying GeoNode\, an open- source geospatial content management system\, in production environments w ithin two Brazilian government agencies: the Geological Survey of Brazil ( SGB) and the Brazilian Federal Police (PF). We'll explore how these agenci es have successfully implemented and customized GeoNode to meet their spec ific needs\, addressing common challenges in large-scale FOSS4G deployment s.\n\nKey points we'll cover:\n\n1. SGB's approach:\n - Developing a Hel m chart for automated GeoNode 4 installation on Red Hat OpenShift\n - Ad dressing security requirements like rootless execution and random UID supp ort\n - Implementing autoscaling for most components based on CPU and me mory utilization\n - Exploring cluster implementation of GeoServer for i mproved scalability\n\n2. PF's customizations:\n - Creating a dedicated "inteligeo-deploy" repository for enhanced deployment features\n - Imple menting centralized configuration and logging\n - Improving security by separating credentials and using Podman instead of Docker\n - Integratin g with internal systems and scheduling data updates\n\nWe'll discuss the c hallenges faced\, solutions implemented\, and lessons learned from both ap proaches. These case studies demonstrate that FOSS4G solutions like GeoNod e are ready for production use in government agencies\, providing flexibil ity\, scalability\, and security.\n\nBy sharing our experiences\, we aim t o help other organizations successfully deploy GeoNode and other FOSS4G so lutions in production environments. We welcome questions and discussions o n best practices for large-scale FOSS4G implementations. DTSTAMP:20260519T170558Z LOCATION:Room II SUMMARY:Deploying GeoNode in Production: Lessons from Brazilian Government Agencies - Carlos Eduardo Mota\, Daniel Araújo Miranda URL:https://talks.staging.osgeo.org/foss4g-2024/talk/BCYFPL/ END:VEVENT BEGIN:VEVENT UID:pretalx-foss4g-2024-3MTN3L@talks.staging.osgeo.org DTSTART;TZID=-03:20241206T160000 DTEND;TZID=-03:20241206T163000 DESCRIPTION:The Geological Survey of Brazil (SGB) is under a digital transf ormation process. One of the pillars of this process involves speed\, scal ability\, security and availability of data produced. Furthermore\, CPRM i s creating a favorable environment for the adoption of cloud-based archite cture.\n\nThis paper aims to present an overview of a new developed Geolog ical Spatial Data Infrastructure for the SGB. The GeoSGB is the main sourc e for internal mapping\, infographic and dashboard applications. It will a ssimilate legacy services\, that runs on isolated map servers\, such as se lf hosted node of OneGeology OGC Services.\n\nThe solution adopted is GeoN ode 4.2\, which brings together map\, data services\, metadata catalog and spatial database. It is free and open source software with a very active community. In addition\, GeoNode has a good content management system\, a rich API\, and it’s fully customizable. To meet data access demands\, it was customized to run in Kubernetes-based environments and each mapped ar ea produces its own geoservice\, exportable to different formats\, such as shapefile and geotiff.\n\nHowever\, it became necessary a complete separa tion between the production and publishing environments. SGB’s productio n pipeline is composed by internally developed data management software. S ome of these systems are being modernized\, with updates on business rules \, frameworks and security. GIS work is carried out in ArcGIS Enterprise® \, with some exceptions in QGIS and GeoServer. With this background\, it s hould be considered as a hybrid GIS model.\n\nAbout database structures\, a process of harmonization was necessary\, mainly those produced from prop rietary GIS. For legacy reasons\, the proprietary structures were maintain ed\, as long as possible to export to OGC WKT or WKB. Exported geometries are analyzed for compliance with Simple Features Standard (OGC/ISO19125). The information eligible for publication were consolidated in database vie ws and is literally replicated to GeoSGB\, by script.\n\n\nThe metadata pr oduction for continuous databases is carried out semi-automatically – te mplated - in accordance with the mapping program. This is possible by inte grating GeoNode's APIs with internal databases\, delivering associated met adata and resources directly to the authors. The contact with (meta)data a uthors were managed by GeoNode.\n\nThe symbolization of thematic layers in volved the development of interoperable libraries\, based on SVG glyphs in serted in OpenType fonts (ISO/IEC 14496-22:2007)\, with near equal renderi ng among different multi-platform GIS software.\n\nData and metadata pipel ines were implemented using Python scripts\, with specific libraries assoc iated with GeoNode APIs. Apache Airflow manages the entire process of extr acting internal bases\, quality tests\, structure analysis and loading on the GeoSGB database server\, including being responsible for notification activities.\n\nSo\, GeoSGB now is a continuous development platform\, with focus in increase quality in delivered data to customers.\n\nThe future p erspectives involve the transformation itself into research line in geotec hnologies and high-performance IT services. It shoud envolve plug-in devel opment for data management\, processing and visualization including use of artificial intelligence. In operational terms\, adoption of OGC APIs\, da ta internationalization and harmonization\, associated with adoption of OG C specific standards\, such as GeoSciML and WaterML contributes to become SGB a global supplier of geoscientific data. DTSTAMP:20260519T170558Z LOCATION:Room I SUMMARY:OpenGeoSGB: State of the art of Transition to an Open Source\, semi -automated\, FAIR-ready Geological Spatial Data Infrastructure of the Geol ogical Survey of Brazil - Carlos Eduardo Mota URL:https://talks.staging.osgeo.org/foss4g-2024/talk/3MTN3L/ END:VEVENT END:VCALENDAR