Scribd Logo
Hochladen

Willkommen bei Scribd!

  • Apache Nifi - Consume Kafka + Merge Content + Put HDFS To Avoid Small Files

    Hochgeladen von

    Csvv Vardhan
    27 Ansichten1 Seite
    The document discusses using Apache NiFi to consume messages from a Kafka topic containing over 2 million messages. Putting the messages directly to HDFS using the PutHDFS processor generates many small files. To avoid this, the document suggests using the Merge Content processor before PutHDFS to combine messages and write fewer, larger files to HDFS. Help is requested to determine if any configuration changes are needed when scaling to very large topic sizes.

    Originalbeschreibung:

    nifi

    Originaltitel

    Consume Kafka

    Copyright

    © © All Rights Reserved

    Verfügbare Formate

    PDF, TXT oder online auf Scribd lesen

    Stufen Sie dieses Dokument als nützlich ein?

    Sind diese Inhalte unangemessen?

    Dieses Dokument melden
    The document discusses using Apache NiFi to consume messages from a Kafka topic containing over 2 million messages. Putting the messages directly to HDFS using the PutHDFS processor generates many small files. To avoid this, the document suggests using the Merge Content processor before PutHDFS to combine messages and write fewer, larger files to HDFS. Help is requested to determine if any configuration changes are needed when scaling to very large topic sizes.

    Copyright:

    © All Rights Reserved
    Als PDF, TXT herunterladen oder online auf Scribd lesen
    Markieren Sie unangemessene Inhalte
    27 Ansichten1 Seite

    Apache Nifi - Consume Kafka + Merge Content + Put HDFS To Avoid Small Files

    Hochgeladen von

    Csvv Vardhan
    The document discusses using Apache NiFi to consume messages from a Kafka topic containing over 2 million messages. Putting the messages directly to HDFS using the PutHDFS processor generates many small files. To avoid this, the document suggests using the Merge Content processor before PutHDFS to combine messages and write fewer, larger files to HDFS. Help is requested to determine if any configuration changes are needed when scaling to very large topic sizes.

    Copyright:

    © All Rights Reserved
    Als PDF, TXT herunterladen oder online auf Scribd lesen
    Markieren Sie unangemessene Inhalte
    von 1

    hortonworks dataflow - Apache Nifi - Consume Kafka + Merge Content ... https://stackoverflow.com/questions/51403511/apache-nifi-consume-kaf...

    How are we doing? Please help us improve Stack Overflow. Take our short survey

    Apache Nifi - Consume Kafka + Merge Content + Put


    HDFS to avoid small files
    I am having around 2000000 messages in Kafka topic and I want to put these records into HDFS
    using NiFi,so I am using PutHDFS processor for this along with ConsumeKafka_0_10 but it
    2 generates small files in HDFS, So I am using Merge Content processor for the merging the
    records before pushing the file.

    Please help if the configuration needs changes This works fine for small number of messages
    but writes a single file for every record when it comes to topics with massive data.

    Thank you!!

    apache-nifi hortonworks-dataflow

    edited Jul 19 '18 at 7:17 asked Jul 18 '18 at 13:39


    Sivaprasanna BARATH
    Sethuraman 87 9
    2,714 1 17 44

    1 Answer

    By using our site, you acknowledge that you have read and understand our Cookie Policy, Privacy Policy, and
    our Terms of Service.

    1 of 1 24-06-2019, 11:49 pm

    Das könnte Ihnen auch gefallen