Network Information Processing Analysis Based on Big Data Parallel Graph Partitioning Algorithm

Abstract

This research proposes an efficient parallel graph partitioning algorithm for the big data environment, aiming to solve the bottlenecks of traditional clustering techniques in terms of processing speed and scalability. The algorithm adopts a multi-level graph partitioning framework, decomposing the network information processing task into multiple levels, gradually simplifying the graph structure and backtracking refinement, thereby significantly reducing the computational complexity while ensuring the partitioning quality. The algorithm focuses on balancing the node cohesion within partitions and the edge cutting cost of inter-partition communication. By constructing a global objective function, it minimizes the number of edges across partitions and the workload differences among various sub-graphs, thereby achieving a more balanced partitioning result. The research results show that this algorithm achieves a resource utilization rate of 0.95. In the Hadoop cluster environment, 95% of the computing resources are effectively used for actual task processing, which is significantly higher than that of the competing algorithms. The energy efficiency ratio reaches 0.98, indicating that the number of tasks completed per unit of energy consumption is close to the optimal level, which is superior to the 0.78 to 0.67 range of existing methods, reflecting the advantages of this algorithm in green computing. The load imbalance rate is only 0.00395, and the point weight imbalance rate is 0.00141, which are much lower values than those of the comparison algorithm. This indicates that the algorithm achieves a high degree of balance in task allocation and node weight distribution, effectively avoiding resource waste and performance bottlenecks.