Update "Ingest logs from a Python application using Filebeat”

[vishaangelova]

In this PR:

• Restructured document to follow a more logical structure and flow
• Removed (unnecessary) references to ECE
• Updated outdated info related to UI changes
• Updated outdated info related to the Filebeat config: replaced deprecated log input type with the filestream input type and adjusted the config options accordingly
• Updated the screenshots (removed outdated Donut chart)
• Promoted bold “titles” of sections to headers
• Fixed formatting, typos, and style issues

Closes #1640 and #125444

Preview

manage-data/ingest/ingesting-data-from-applications/ingest-logs-from-python-application-using-filebeat.md

[github-actions]

🔍 Preview links for changed docs:

• manage-data/ingest/ingesting-data-from-applications/ingest-logs-from-python-application-using-filebeat.md

🔔 The preview site may take up to 3 minutes to finish building. These links will become live once it completes.

[theletterf]

First pass — nice work!

[eedugon]

I like (a lot!) and agree with how we are updating and improving this document, but I'd like to share some high level thoughts about it.

• Why are we still mentioning specifically ECH as the Elastic Stack destination? The doc should be valid to send data to a self-managed, ECK or ECE managed cluster also, and most probably also valid for serverless (this might require a bit extra investigation, as maybe in Serverless we rely only on api keys for authentication and not user/passwords anymore).

• Anyway, shouldn't we at least mention that the doc example uses ECH as the Elastic Stack destination but it could be adapted to other deployment types such as ECK, ECE or self-managed? (cc: @shainaraskas , in the past we have found similar occurrences of this type of issue, where a doc was originally written towards one deployment type but there wasn't any restriction in reality).

IMO, the step Create an Elastic Cloud Hosted deployment would fit better as a pre-requisite instead of an actual step, as I'd say that the majority of times a reader needs to learn how to send data to Elasticsearch they already have an Elasticsearch cluster somewhere. As a step the procedure doesn't flow well, as we should be focusing on Python, Filebeat, connectivity + authentication, and configuration (the other steps) more than creating the deployment (which obbviously should exist). I would suggest to add a short pre-requisites section identifying that the user needs to have an available Elastic deployment with at least Elasticsearch and Kibana, and they could create a ECH deployment (link to the official doc) if they need.

The next step Connect securely to Elastic Cloud Hosted could be renamed to something like Preparing connection and authentication details. The step itself is deployment agnostic (except the cloudID thing that is only available in ECE and Elastic Cloud), and it's all about preparing & gathering the authentication details to be used in Filebeat more than actually connecting. That change would also help to decouple the guide from ECH.

If we want to make the doc deployment agnostic (or almost agnostic), the filebeat configuration section should also mention that if the destination is not Elastic Cloud or ECE, the user should use the actual URL of the Elasicsearch cluster instead of cloud.id -> But for this the best would also be to link to the official docs, using these two links:

https://www.elastic.co/docs/reference/beats/filebeat/configure-cloud-id -> for ECH and ECE only
https://www.elastic.co/docs/reference/beats/filebeat/elasticsearch-output -> For all deployment types (including ECH and ECE too)

cc: @georgewallace : I think this type of feedback is related with your suggestions about reviews, right? trying to make the new IA and guides more consistent with the new approach.

[vishaangelova]

@eedugon thank you, that’s some really good feedback! I appreciate it!

I agree that the gist of Create an Elastic Cloud Hosted deployment would fit better as part of the prerequisites - I don’t know why I didn’t think of this earlier, it makes perfect sense.

I also agree with your suggestion to rename Connect securely to Elastic Cloud Hosted - I’d probably rephrase to Prepare your connection and authentication details to remain consistent with the heading structure.

I think you’re right about making this doc deployment-agnostic, but I’m also wary of how this may impact the others steps in this guide. For example, the difference in authentication, as you mentioned, or the possible UI differences (e.g., in the menu structure) which could impact the accuracy of the detailed steps for creating visualizations... What I like in the guide’s current form is that it describes a workflow from start to finish, and you can actually see a working example of Filebeat for log ingest - whereas if you detach it from a specific deployment, it gets much harder for the reader to troubleshoot possible hiccups that may arise from their specific deployment context. So making this deployment-agnostic would require testing the steps in multiple contexts to make sure they are accurate for all deployments.

If @shainaraskas and @colleenmcginnis agree that we should make this doc deployment-agnostic, we could create a follow-up issue for that purpose specifically, and work on it separately. What do you think?

That said, I really like the idea of adding a note that this doc uses ECH as the Elastic Stack destination but that it could be adapted to other deployment types. I’m going to add this to the doc.

[eedugon]

@vishaangelova , I'd say there's a middle point between making a guide like this completely deployment agnostic vs being written only with ECH deployments in mind.

Originally this doc was a Getting started guide in Elastic Cloud documentation. Now we have the SAME document in a MANAGE DATA -> Ingestion -> Ingesting data from applications section of the new IA, which is generic and deployment agnostic -> That makes the doc to not fit well.

We have migrated thousands of documents that had a very narrowed scope to the new IA, and in a lot of cases we ended up with leftovers like this.

We need to improve this type of issue whenever we start improving and giving some love to documents, as with the new IA in mind the scope of this document talking about Elastic Cloud Hosted deployments when it's really about Ingest logs from a Python application using Filebeat in the Manage Data section feels wrong and incorrect.

And I'm not saying to make it deployment agnostic right away, but if the guide is not really related with ECH and doesn't strictly need a ECH deployment we should just mention that at the beginning of the document, with some kind of note saying that the guide uses a ECH deployment as the destination of the data but it could be adapted to any other deployment type. That would be at least the easiest solution until someone decides to make it fully deployment agnostic (which is how it should be).

I would say that we shouldn't move forward with improving this document without considering that, as this document feels wrong in the new IA (of course it's not a problem of this PR, it was the original doc itself). Just a minor note would at least make it digestible in the way it is.

As a separate topic (not deployment types related), I'm also missing in the doc maybe some useful glue / links to:

• Filebeat installation and setup --> this doc / guide shows a way to install (actually download and directly run) filebeat but filebeat can be installed with package managers and in multiple ways, so a link to the official filebeat installation & setup doc would be beneficial and would make the doc to flow better with the rest of the documentation we have (currently it feels like a silo where we are duplicating part of the info we have in other places).

• Filebeat configuration --> This one is nicely covered for example where we offer the link to the ndjson parser. If there are other useful parts where we should link to the official config doc for the user to know more about filebeat configuration we should consider it.

• Links to authentication and authorization for users to get more details about the different auth methods showed in the guide (cloud ID + auth / API Keys).

@shainaraskas , I'd like to hear your thoughts if possible, because this doc feels like the old end to end guides that were duplicating content in certain sections. For these docs to evolve better I think they should be better linked (to and from) with the rest of the documentation.

[eedugon]

Another possibility, we merge this PR as it is, and admin-docs / control-plane docs (so myself for example :) ) could take care of working on the deployment type side of things + adding links to other parts of the deploy and manage docs when needed.

Whatever we feel!

[vishaangelova]

I’ve added a note at the beginning that the steps in this guide could be adapted for other deployments, but I kept the references to ECH for now. Let’s see what the others say about how to proceed with removing those. I’ve also restructured the doc a bit to reflect the observations in your first comment.

You are right about the cross-links, I can add references to the docs you mentioned.

[theletterf]

Not an SME, but writeup LGTM!

[eedugon]

@vishaangelova , I think it looks very nice now! I'll add some minor comments in other parts trying to enrich part of the content with useful links and alternatives (cloud ID for example).

[shainaraskas]

closing the loop because I was tagged: I agree that adding a note about the tutorial being written for ECH but being valid with small changes for other deployment types is a great temporary patch.

I think adding an item in our backlog to ultimately make this tutorial generic is a good idea - they might be low priority, but they're good to track.

[eedugon]

LGTM! I've added some minor comments suggesting some extra links and clarifications that would help for other deployment types.