Hi guys,
I've no ideia of what could be the cause of this error. On TD Docs (http://www.info.teradata.com/htmlpubs/DB_TTU_14_10/index.html#page/General_Reference/B035_1096_112K/TPT.41.1267.html), the explanation doesn't say how to solve the problem.
The thing is: I wrote a vbs script to generate a TPT script and a amj file to migrate a entire SQLServer database to TD. After several adjustements, I've reached a script that works very well, but it can generate pretty big tpt scripts. The one that is causing me the problem has almost 14k lines. Although the script is big, it is simple. Basicly, a DDL Operator, then, one schema per table. After, one Dataconector Producer (access module OLEDB_AXSMOD) per table (each one references a job in the .amj file) and at last, one Load Operator per table. Toghether with the script, I also create a amj file with one job per table (acctually, a select statement wich I use to do some transformations).
On the steps, first I call the DDL Operator with some DDL Instructions to drop and create the stage tables. Then, the sptes to pull the data from SQLServer to TD (one per table) and push into TD, and at the end, a last call to the DDL Operator to rename the normal tables to backup tables, and the stage tables to the official tables.
As I've said before, the generated script can get a little big, with hundreds of steps. Then, with this script with 257 steps (apart from the initial and final step), when it gets about the 90th I get the following message: "TPT_INFRA TPT01057 Error Insufficient main storage for attempted allocation".
That is the end. Tpt hangs and I can't get anywere from there. There isn't a exactly point where it hangs. It can be at the 90th, 91th, 92th and etc. If I ctrl+c the job e reissue the tbuild command, Tpt processes one more step and again issues the TPT01057 message. The tpt infra doesn't responde to commands like twbcmd <job> resume nor twbcmd <job> pause (still it says that is processing the command). The twbstat says that the job is running.
One important detail is: this only happens between the end of a step and the start of the next step. In other words, it doesn't happens in the middle of a step.
This is one example of the output from tbuild:
Job step step_91 completed successfully
Teradata Parallel Transporter src_operator_ativa_ModeloPapelTrabalho_TipoOs: TPT19006 Version 14.10.00.01
src_operator_ativa_ModeloPapelTrabalho_TipoOs Instance 1 directing private log report to 'producer_log-1'.
src_operator_ativa_ModeloPapelTrabalho_TipoOs: TPT19003 NotifyMethod: 'None (default)'
src_operator_ativa_ModeloPapelTrabalho_TipoOs: TPT19008 DataConnector Producer operator Instances: 1
Teradata Parallel Transporter Load Operator Version 14.10.00.01
load_operator_ativa_ModeloPapelTrabalho_TipoOs: private log specified: load_log_ativa_ModeloPapelTrabalho_TipoOs
src_operator_ativa_ModeloPapelTrabalho_TipoOs: TPT19003 ECI operator ID: src_operator_ativa_ModeloPapelTrabalho_TipoOs-4624
load_operator_ativa_ModeloPapelTrabalho_TipoOs: connecting sessions
src_operator_ativa_ModeloPapelTrabalho_TipoOs: TPT19222 Operator instance 1 processing file 'corp_ativa_acesso.amj'.
load_operator_ativa_ModeloPapelTrabalho_TipoOs: preparing target table
load_operator_ativa_ModeloPapelTrabalho_TipoOs: entering Acquisition Phase
load_operator_ativa_ModeloPapelTrabalho_TipoOs: entering Application Phase
load_operator_ativa_ModeloPapelTrabalho_TipoOs: Statistics for Target Table: 'Desenvolvimento.bdc_carga_ativa_ModeloPapelTra
balho_TipoOs'
load_operator_ativa_ModeloPapelTrabalho_TipoOs: Total Rows Sent To RDBMS: 99
load_operator_ativa_ModeloPapelTrabalho_TipoOs: Total Rows Applied: 99
load_operator_ativa_ModeloPapelTrabalho_TipoOs: Total Rows in Error Table 1: 0
load_operator_ativa_ModeloPapelTrabalho_TipoOs: Total Rows in Error Table 2: 0
load_operator_ativa_ModeloPapelTrabalho_TipoOs: Total Duplicate Rows: 0
src_operator_ativa_ModeloPapelTrabalho_TipoOs: TPT19221 Total files processed: 1.
load_operator_ativa_ModeloPapelTrabalho_TipoOs: disconnecting sessions
load_operator_ativa_ModeloPapelTrabalho_TipoOs: Total processor time used = '1.46641 Second(s)'
load_operator_ativa_ModeloPapelTrabalho_TipoOs: Start : Mon Aug 18 18:53:19 2014
load_operator_ativa_ModeloPapelTrabalho_TipoOs: End : Mon Aug 18 18:53:30 2014
Job step step_92 completed successfully
TPT_INFRA: TPT01057: Error: Insufficient main storage for attempted allocation
As you can see, after TPT completed the 92th step, it hanged. From tlogview:
Task(SELECT_2[0001]): checkpoint completed, status = Success
Task(APPLY_1[0004]): checkpoint completed, status = Success
Task(APPLY_1[0003]): checkpoint completed, status = Success
Task(APPLY_1[0001]): checkpoint completed, status = Success
Task(APPLY_1[0002]): checkpoint completed, status = Success
load_operator_ativa_ModeloPapelTrabalho_TipoOs: entering Application Phase
load_operator_ativa_ModeloPapelTrabalho_TipoOs: Statistics for Target Table: 'Desenvolvimento.bdc_carga_ativa_ModeloPapelTra
balho_TipoOs'
load_operator_ativa_ModeloPapelTrabalho_TipoOs: Total Rows Sent To RDBMS: 99
load_operator_ativa_ModeloPapelTrabalho_TipoOs: Total Rows Applied: 99
load_operator_ativa_ModeloPapelTrabalho_TipoOs: Total Rows in Error Table 1: 0
load_operator_ativa_ModeloPapelTrabalho_TipoOs: Total Rows in Error Table 2: 0
load_operator_ativa_ModeloPapelTrabalho_TipoOs: Total Duplicate Rows: 0
TPT_INFRA: TPT02255: Message Buffers Sent/Received = 1, Total Rows Received = 0, Total Rows Sent = 0
TPT_INFRA: TPT02255: Message Buffers Sent/Received = 0, Total Rows Received = 0, Total Rows Sent = 0
TPT_INFRA: TPT02255: Message Buffers Sent/Received = 0, Total Rows Received = 0, Total Rows Sent = 0
TPT_INFRA: TPT02255: Message Buffers Sent/Received = 0, Total Rows Received = 0, Total Rows Sent = 0
TPT_INFRA: TPT02255: Message Buffers Sent/Received = 1, Total Rows Received = 0, Total Rows Sent = 0
src_operator_ativa_ModeloPapelTrabalho_TipoOs: TPT19221 Total files processed: 1.
load_operator_ativa_ModeloPapelTrabalho_TipoOs: disconnecting sessions
load_operator_ativa_ModeloPapelTrabalho_TipoOs: Total processor time used = '1.46641 Second(s)'
load_operator_ativa_ModeloPapelTrabalho_TipoOs: Start : Mon Aug 18 18:53:19 2014
load_operator_ativa_ModeloPapelTrabalho_TipoOs: End : Mon Aug 18 18:53:30 2014
Job step step_92 completed successfully
TPT_INFRA: TPT01057: Error: Insufficient main storage for attempted allocation
TPT_INFRA: TPT02813: Error: Failed to create the Job variable
Teradata Parallel Transporter Executor Version 14.10.00.01
Teradata Parallel Transporter Coordinator Version 14.10.00.01
Teradata Parallel Transporter Executor Version 14.10.00.01
Teradata Parallel Transporter Executor Version 14.10.00.01
Teradata Parallel Transporter Executor Version 14.10.00.01
Teradata Parallel Transporter Executor Version 14.10.00.01
From the tpt script, here is a small extract (only things related with the step 92 and 93). Again, as I've said before, it hangs after the 90th, and not necessarily between the 91th and 92th.
DEFINE JOB FILE_LOAD
DESCRIPTION 'Job corp_ativa_acesso'
(
DEFINE OPERATOR DDL_Operator
TYPE DDL
ATTRIBUTES (
VARCHAR PrivateLogName = 'ddl_log',
VARCHAR UserName = 'DBADMIN',
VARCHAR UserPassword = '*********',
VARCHAR ARRAY ErrorList = ['3807','3803'],
VARCHAR TdpId = 'maxcgu01-1-1'
);
DEFINE SCHEMA schema_ativa_ModeloPapelTrabalho_TipoOs (
"IdModeloPapelTrabalho_TipoOs" INTEGER,"IdModeloPapelTrabalho" INTEGER,"IdTipoOs" INTEGER,"BolObrigatorio" SMALLINT,"IdPerfil" INTEGER
);
DEFINE SCHEMA schema_ativa_ModeloPapelTrabalhoVinculado (
"IdModeloPapelTrabalhoVinculado" INTEGER,"IdModeloPapelTrabalhoA" INTEGER,"IdModeloPapelTrabalhoB" INTEGER
);
DEFINE OPERATOR src_operator_ativa_ModeloPapelTrabalho_TipoOs
DESCRIPTION 'Odbc Operator para tabela [ativa].[ModeloPapelTrabalho_TipoOs]'
TYPE DATACONNECTOR PRODUCER
SCHEMA schema_ativa_ModeloPapelTrabalho_TipoOs
ATTRIBUTES (
VARCHAR AccessModuleName = 'OLEDB_AXSMOD',
VARCHAR FileName = 'corp_ativa_acesso.amj',
VARCHAR Format = 'Formatted',
VARCHAR AccessModuleInitStr = 'noprompt jobid=92',
VARCHAR OpenMode = 'Read',
VARCHAR EnableScan = 'No',
VARCHAR IndicatorMode = 'Yes',
VARCHAR PrivateLogName = 'producer_log'
);
DEFINE OPERATOR src_operator_ativa_ModeloPapelTrabalhoVinculado
DESCRIPTION 'Odbc Operator para tabela [ativa].[ModeloPapelTrabalhoVinculado]'
TYPE DATACONNECTOR PRODUCER
SCHEMA schema_ativa_ModeloPapelTrabalhoVinculado
ATTRIBUTES (
VARCHAR AccessModuleName = 'OLEDB_AXSMOD',
VARCHAR FileName = 'corp_ativa_acesso.amj',
VARCHAR Format = 'Formatted',
VARCHAR AccessModuleInitStr = 'noprompt jobid=93',
VARCHAR OpenMode = 'Read',
VARCHAR EnableScan = 'No',
VARCHAR IndicatorMode = 'Yes',
VARCHAR PrivateLogName = 'producer_log'
);
DEFINE OPERATOR load_operator_ativa_ModeloPapelTrabalho_TipoOs
TYPE LOAD
SCHEMA schema_ativa_ModeloPapelTrabalho_TipoOs
ATTRIBUTES(
VARCHAR PrivateLogName = 'load_log_ativa_ModeloPapelTrabalho_TipoOs',
VARCHAR UserName = 'DBADMIN',
VARCHAR UserPassword = '*****',
VARCHAR TdpId = 'maxcgu01-1-1',
VARCHAR TargetTable = 'Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalho_TipoOs',
VARCHAR LogTable = 'Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalho_TipoOs_log',
VARCHAR ErrorTable1 = 'Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalho_TipoOs_error_1',
VARCHAR ErrorTable2 = 'Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalho_TipoOs_error_2'
);
DEFINE OPERATOR load_operator_ativa_ModeloPapelTrabalhoVinculado
TYPE LOAD
SCHEMA schema_ativa_ModeloPapelTrabalhoVinculado
ATTRIBUTES(
VARCHAR PrivateLogName = 'load_log_ativa_ModeloPapelTrabalhoVinculado',
VARCHAR UserName = 'DBADMIN',
VARCHAR UserPassword = '******',
VARCHAR TdpId = 'maxcgu01-1-1',
VARCHAR TargetTable = 'Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalhoVinculado',
VARCHAR LogTable = 'Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalhoVinculado_log',
VARCHAR ErrorTable1 = 'Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalhoVinculado_error_1',
VARCHAR ErrorTable2 = 'Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalhoVinculado_error_2'
);
STEP step_inicial_1 (
APPLY
('drop table Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalho_TipoOs;'),
('drop table Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalho_TipoOs_log'),
('drop table Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalho_TipoOs_error_1'),
('drop table Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalho_TipoOs_error_2'),
('create table Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalho_TipoOs(
IdModeloPapelTrabalho_TipoOs INTEGER,IdModeloPapelTrabalho INTEGER,IdTipoOs INTEGER,BolObrigatorio SMALLINT,IdPerfil INTEGER
)UNIQUE PRIMARY INDEX(IdModeloPapelTrabalho_TipoOs)')
,
('drop table Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalhoVinculado;'),
('drop table Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalhoVinculado_log'),
('drop table Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalhoVinculado_error_1'),
('drop table Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalhoVinculado_error_2'),
('create table Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalhoVinculado(
IdModeloPapelTrabalhoVinculado INTEGER,IdModeloPapelTrabalhoA INTEGER,IdModeloPapelTrabalhoB INTEGER
)UNIQUE PRIMARY INDEX(IdModeloPapelTrabalhoVinculado)')
TO OPERATOR (DDL_Operator);
);
STEP step_92 (
APPLY ('Insert into Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalho_TipoOs ("IdModeloPapelTrabalho_TipoOs","IdModeloPapelTrabalho","IdTipoOs","BolObrigatorio","IdPerfil") values (:"IdModeloPapelTrabalho_TipoOs",:"IdModeloPapelTrabalho",:"IdTipoOs",:"BolObrigatorio",:"IdPerfil")')
TO OPERATOR (load_operator_ativa_ModeloPapelTrabalho_TipoOs[4])
SELECT "IdModeloPapelTrabalho_TipoOs","IdModeloPapelTrabalho","IdTipoOs","BolObrigatorio","IdPerfil" FROM OPERATOR (src_operator_ativa_ModeloPapelTrabalho_TipoOs);
);
STEP step_93 (
APPLY ('Insert into Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalhoVinculado ("IdModeloPapelTrabalhoVinculado","IdModeloPapelTrabalhoA","IdModeloPapelTrabalhoB") values (:"IdModeloPapelTrabalhoVinculado",:"IdModeloPapelTrabalhoA",:"IdModeloPapelTrabalhoB")')
TO OPERATOR (load_operator_ativa_ModeloPapelTrabalhoVinculado[4])
SELECT "IdModeloPapelTrabalhoVinculado","IdModeloPapelTrabalhoA","IdModeloPapelTrabalhoB" FROM OPERATOR (src_operator_ativa_ModeloPapelTrabalhoVinculado);
);
STEP step_final_1 (
APPLY
('drop table Desenvolvimento.bdc_ativa_ModeloPapelTrabalho_TipoOs_bkp;'),
('rename table Desenvolvimento.bdc_ativa_ModeloPapelTrabalho_TipoOs to Desenvolvimento.bdc_ativa_ModeloPapelTrabalho_TipoOs_bkp;'),
('rename table Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalho_TipoOs to Desenvolvimento.bdc_ativa_ModeloPapelTrabalho_TipoOs;'),
('drop table Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalho_TipoOs_log'),
('drop table Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalho_TipoOs_error_1'),
('drop table Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalho_TipoOs_error_2')
,
('drop table Desenvolvimento.bdc_ativa_ModeloPapelTrabalhoVinculado_bkp;'),
('rename table Desenvolvimento.bdc_ativa_ModeloPapelTrabalhoVinculado to Desenvolvimento.bdc_ativa_ModeloPapelTrabalhoVinculado_bkp;'),
('rename table Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalhoVinculado to Desenvolvimento.bdc_ativa_ModeloPapelTrabalhoVinculado;'),
('drop table Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalhoVinculado_log'),
('drop table Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalhoVinculado_error_1'),
('drop table Desenvolvimento.bdc_carga_ativa_ModeloPapelTrabalhoVinculado_error_2')
TO OPERATOR (DDL_Operator);
);
);
And at last, a piece of the .amj file:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?OLE_DB_AXSMOD_FirstCompatibleVersion 14.00.00.00?>
<!--Configuration information for the OLE DB AXSMOD-->
<OLE_DB_AXSMOD_Jobs>
<Job Id="92">
<Source>
<DataSourceParseName>{397C2819-8272-4532-AD3A-FB5E43BEAA39}<!--SQL Server Native Client 11.0 (SQLNCLI11)-->
</DataSourceParseName>
<DataSourceProperties>
<PropertySet>
...
</PropertySet>
</DataSourceProperties>
<TableCommand>SELECT [IdModeloPapelTrabalho_TipoOs],[IdModeloPapelTrabalho],[IdTipoOs],convert(smallint,[BolObrigatorio]) as [BolObrigatorio],[IdPerfil] FROM [ativa].[ModeloPapelTrabalho_TipoOs]</TableCommand>
<Columns>
<Column><Selected/><SourceName>IdModeloPapelTrabalho_TipoOs</SourceName><DestinationName>IdModeloPapelTrabalho_TipoOs</DestinationName><TypeName>INTEGER</TypeName></Column>
<Column><Selected/><SourceName>IdModeloPapelTrabalho</SourceName><DestinationName>IdModeloPapelTrabalho</DestinationName><TypeName>INTEGER</TypeName></Column>
<Column><Selected/><SourceName>IdTipoOs</SourceName><DestinationName>IdTipoOs</DestinationName><TypeName>INTEGER</TypeName></Column>
<Column><Selected/><SourceName>BolObrigatorio</SourceName><DestinationName>BolObrigatorio</DestinationName><TypeName>SMALLINT</TypeName></Column>
<Column><Selected/><SourceName>IdPerfil</SourceName><DestinationName>IdPerfil</DestinationName><TypeName>INTEGER</TypeName></Column>
</Columns>
<LocationOfLogTables>0<!--User's default database 0, Source database 1, Other database 2-->
</LocationOfLogTables>
<OtherDatabase/>
<CharDataTransferUTF8>1<!--OldMethod 0, NewMethod 1-->
</CharDataTransferUTF8>
</Source>
<CharacterEncoding>ASCII</CharacterEncoding>
<CheckpointInterval/>
<LargeDecimalSupport>Supported</LargeDecimalSupport>
<RowsPerFetch>15000</RowsPerFetch>
<BufferSize/>
</Job>
<Job Id="93">
<Source>
<DataSourceParseName>{397C2819-8272-4532-AD3A-FB5E43BEAA39}<!--SQL Server Native Client 11.0 (SQLNCLI11)-->
</DataSourceParseName>
<DataSourceProperties>
<PropertySet>
...
</PropertySet>
</DataSourceProperties>
<TableCommand>SELECT [IdModeloPapelTrabalhoVinculado],[IdModeloPapelTrabalhoA],[IdModeloPapelTrabalhoB] FROM [ativa].[ModeloPapelTrabalhoVinculado]</TableCommand>
<Columns>
<Column><Selected/><SourceName>IdModeloPapelTrabalhoVinculado</SourceName><DestinationName>IdModeloPapelTrabalhoVinculado</DestinationName><TypeName>INTEGER</TypeName></Column>
<Column><Selected/><SourceName>IdModeloPapelTrabalhoA</SourceName><DestinationName>IdModeloPapelTrabalhoA</DestinationName><TypeName>INTEGER</TypeName></Column>
<Column><Selected/><SourceName>IdModeloPapelTrabalhoB</SourceName><DestinationName>IdModeloPapelTrabalhoB</DestinationName><TypeName>INTEGER</TypeName></Column>
</Columns>
<LocationOfLogTables>0<!--User's default database 0, Source database 1, Other database 2-->
</LocationOfLogTables>
<OtherDatabase/>
<CharDataTransferUTF8>1<!--OldMethod 0, NewMethod 1-->
</CharDataTransferUTF8>
</Source>
<CharacterEncoding>ASCII</CharacterEncoding>
<CheckpointInterval/>
<LargeDecimalSupport>Supported</LargeDecimalSupport>
<RowsPerFetch>15000</RowsPerFetch>
<BufferSize/>
</Job>
</OLE_DB_AXSMOD_Jobs>
Any ideas?