Log Analysis

Log Analysis helps quickly create an end-to-end log analysis platform covering data collection, cleansing, retrieval, reporting, and display. Get the comprehensive insight you need from log data.
Solution Advantages
  • Ease of Use

    This solution enables you to quickly create an end-to-end log analysis platform without the need for programming.

  • Easy O&M and Low Cost

    Most of the EI services involved in this solution adopt the serverless architecture or are fully managed, freeing your from time-intensive O&M. Data Ingestion Service (DIS), Data Lake Insight (DLI), and Object Storage Service (OBS) adopt the pay-per-use billing mode, which means you only need to pay for the resources you use.

  • Easy Interconnection

    Standard APIs of open-source applications facilitate interconnection with third-party products and downstream applications.

  • Powerful Data Analysis Capabilities

    Now you can perform full-text retrieval on semi-structured text data as well as retrieval and analysis on structured reports.

Business Challenges
  • High Requirements on Log Access Performance

    Logs are stored on different servers and can be collected using various methods, such as Flume, Beats, Rsyslog, SNMP, and customized programs. The cloud platform must be compatible with mainstream log collectors and has high concurrency and low latency requirements so that new logs on multiple servers can be collected in real time. In addition, reliable large-scale data access capabilities are needed for batch importing scenarios.

  • Complex Programming for Log Parsing

    Logs generated by hardware devices, operating systems, commercial software, and self-developed applications are different in formats. Users must spare a lot of time in writing special programs to convert logs of different formats into structured data while ensuring the performance, scalability, stability, and fault recovery in real-time and batch log collection scenarios.

  • Complex Log Analysis Scenarios

    Log analysis is critical in satisfying many business needs. For example, based on log analysis results, you can monitor abnormal metrics to locate system problems during O&M analysis, generate user profiles based on user behaviors to forecast metrics during operation analysis, and detect exceptions and trace the root cause during security audits.

  • High Storage Cost of Historical Data

    Generally, clusters are used to store the ever-increasing amounts of historical data that is infrequently accessed. This means constant addition new nodes when the data volume is nearing the limit.

Typical Scenarios
  • Enterprise Self-built Log Analysis System

  • Interconnection with Third-Party Log Analysis Products

Enterprise Self-built Log Analysis System

Delivers enterprise-class log retrieval and full-stack analysis capabilities.

  1. High Concurrency & Real-Time Access

    The system supports millions of concurrent connections and responds to data requests within milliseconds. Thousands of GB data can be written into a single partition per day.

  2. Multiple Log Collection Modes

    The system supports real-time and batch log collection. It can interconnect with popular collectors like Flume, Beats, and Rsyslog. You can also customize programs to collect logs.

  3. Flexible Log Processing

    The system delivers log collection, parsing, and analysis with high concurrency and low latency and without the need of programming. You can also build applications compatible with Flink and Spark Streaming APIs for real-time analysis.

  4. Separation Between Computing and Storage

    Services related to BI, graph analysis, and machine learning can directly read and write structured data stored in object storage.

  5. O&M Free

    The out-of-the-box services support auto scaling and ensure zero service interruptions.

  6. Comprehensive Log Analysis Capability

    The system offers AI analysis capabilities, supports complex BI analysis, standard SQL, and pre-aggregation. It is fully compatible with Spark APIs and gives you direct access to files in the CSV, JSON, Parquet, and ORC formats in OBS.

  7. Related Services

    Data Ingestion Service

    Cloud Stream Service

    Cloud Search Service

    Data Lake Insight

    Graph Engine Service

    Machine Learning Service

    Object Storage Service

Interconnection with Third-Party Log Analysis Products

Provides standard, low-cost, and O&M-free EI service kernels.

  1. Standard APIs

    Message queues are compatible with Kafka APIs; stream computing services are compatible with the Spark Streaming/Flink APIs; analysis engine services offer the Spark, graph analysis, and AI analysis capabilities; and full-text retrieval is compatible with native Elasticsearch APIs.

  2. Low Cost

    WebScan periodically detects vulnerabilities of cloud servers and web page codes and reminds personnel to address the dangers in a timely manner.

  3. O&M Free

    All EI services capable of log analysis adopt the serverless architecture or are fully managed.

  4. Related Services

    Data Ingestion Service

    Cloud Stream Service

    Cloud Search Service

    Data Lake Insight

    Graph Engine Service

    Machine Learning Service

    Object Storage Service

Recommended Services
  • Cloud Search Service
    Cloud Search Service is a fully hosted distributed search service powered on Elasticsearch. It is fully compatible with Elasticsearch APIs and provides users with structured and unstructured data search, statistics, and report capabilities.
  • Data Lake Insight
    Data Lake Insight (DLI) is a fully managed big data processing and analysis service. With DLI, you can get insights from heterogeneous data of various services on HUAWEI CLOUD by using SQL and Spark programs, without extract, transform, and load (ETL) operations.
  • Cloud Stream Service
    Cloud Stream Service (CS) is a real-time big data stream analysis service running on HUAWEI CLOUD. Compatible with Apache Flink APIs, CS also fully hosts computing clusters, allowing you to focus on StreamSQL services and run jobs in real time.

Create an Account and Experience HUAWEI CLOUD for Free

Register Now