Tag:data file

  • Zip compression / decompression using java


    Due to the limited network bandwidth, the compression of data files is conducive to the rapid transmission of data on the Internet, and also saves Save the external storage space of the server. Java 1.1 implements a single interface between i/o data flow and network data flow, so data compression, network transmission and reconciliation The […]

  • Linux file classification


    catalogue Linux everything is a file Five Linux files Ordinary documents Catalog file Equipment documents Pipeline file Linked file Linux everything is a file Linux abstracts the management of data files, program files, device files, network files, etc. into files, which are managed in a unified way. However, the Linux operating system must also distinguish […]

  • How Oracle sets the tablespace data file size


    The default maximum size of Oracle data file is 32g. If the data file is larger than 32g, it needs to be set at the beginning of database creation.Table space data file capacity and DB_ BLOCK_ Size is related. When the database is initially built, DB_ BLOCK_ The size should be set to 4K, 8K, […]

  • SQL Server batch import big data


    In the past, usually, the method of inserting multiple pieces of data into the database at one time is to insert them one by one with circular code. This method is inefficient in the face of millions or tens of millions of levels of data, and it usually takes a few minutes, or even dozens […]

  • MySQL active / standby setup based on docker


    Docker MySQL master-slave configuration. We will configure MySQL active / standby on a CentOS virtual machine Preliminary planning: Docker MySQL version mysql:5.7 For docker network configuration, we will create a network named MySQL with a subnet mask of Container name corresponding to docker MySQL master: MySQL master, IP address:, mapping port: 3307, file […]

  • [offline computing spark|hive] HDFS small file processing


    background Too many small HDFS files will affect Hadoop scalability and stability, because a large amount of meta information needs to be stored and maintained on the namenode A large number of small files will also lead to poor query analysis performance, because the query engine needs to open / read / close files too […]

  • Iceberg merge small file conflict test


    9b6b5e0d2 (2022-2-9) based on the master branch of iceberg. Parameter description 1、PARTIAL_PROGRESS_ENABLED(partial-progress.enabled)The default is false. This parameter enables consolidation tasks to be submitted in groups. When one of the group tasks fails, the group task can be retried separately. 2、USE_STARTING_SEQUENCE_NUMBER(use-starting-sequence-number)The default is true.This parameter uses the sequencenumber when merging as the sequencenumber of the new […]

  • Who is the Hadoop family of big data development


    The overall architecture of Hadoop Hadoop is an architectural solution for big data. After years of development and evolution, Hadoop has become a huge system. The internal working mechanism of big data training is very complex. It is an overall architecture that combines distributed theory with specific engineering development. The simplest principle of Hadoop is […]

  • [serial] how to master the core technology of opengauss database? Secret 4: handle the transaction mechanism (2)


    catalogueOpengauss database SQL Engine Opengauss database executor Technology Opengauss storage technology Opengauss transaction mechanism Ⅰ. Opengauss database transaction overview Ⅱ. Introduction to acid feature of opengauss transaction 1. Transaction persistence in opengauss2. Transaction atomicity in opengauss3. Transaction consistency in opengauss4. Transaction isolation in opengaussⅢ. Opengauss concurrency controlⅣ. Opengauss distributed transactionOpengauss database security Opengauss transaction mechanism […]

  • MySQL’s faster drop table method for deleting large tables


    MySQL’s faster drop table method for deleting large tables Using hard link and truncate to reduce the impact of drop table on online environment During drop table, all processes, whether DDL or DML, are started by Hang; Continue to execute until the end of drop; This is because InnoDB will maintain a global exclusive lock […]

  • Oracle common maintenance query


    Check deadlock SELECT s.username,l.OBJECT_ID,l.SESSION_ID,s.SERIAL#, l.ORACLE_USERNAME,l.OS_USER_NAME,l.PROCESS FROM V$LOCKED_OBJECT l,V$SESSION S WHERE l.SESSION_ID=S.SID; Force deadlock release alter system kill session ‘1500,6160’; reference resourceswww.jb51.net/article/85039.htm Check the SQL of deadlock and the SQL executed before deadlock according to Sid –Deadlock time select sql_text from v$sqlarea a,v$session b where a.SQL_ID=b.SQL_ID and (b.SID=47 or b.SID=3847); –Before deadlock select sql_text from v$sqlarea […]

  • SQL Server database repair expert sqlrescue


    Focus on the recovery of SQL Server database. Focus on the repair of various blackmail virus encrypted databases. Sqlrescue can be used to recover data from corrupted MS SQL Server database data files (*. MDF *. NDF files). It can save key data and avoid its loss. 1、 The main functions of sqlrescue include: Recovery […]