MACMAP ETL PACKAGES

UPDATED ON 25/09/2012

ETL documentation for the new MAcMap

Copy source files to C:\WMD_DATA\01_Source_DATA\01_MACMAP\01_In for TARIFF and TRADE data.

The format files used in cases where there is bulk insert C:\WMD_DATA\04_Control_files

An Integration Services project is used to ETL the data into new MAcMaps database: WMD datawarehouse.sln

To update tariff data for ITC source only in the ITC Development server, the following files should be executed
in this order:

1. ITC 010 agreement type 01 into T_AGREEMENT_TYPE.sql

2. ITC 020 agreement 01 into WMDS tariff_agreement.dtsx
3. ITC 020 agreement 02 into R_WORD.sql
4. ITC 020 agreement 03 into R_TRANSLATION.sql
5. ITC 020 agreement 04 into T_AGREEMENT.sql
6. ITC 030 partner agreement 01 into WMDS partner_agreement.dtsx
7. ITC 030 partner agreement 02 into T_CONTRACT.dtsx
8. ITC 040 ntlc description 01 into WMDS tar_ntlc_description.dtsx
9. ITC 040 ntlc description 02 into T_NTLC.dtsx
10. ITC 050 tariffs 01 into WMDS tar_ntlc.dtsx
11. ITC 050 tariffs 03 into T_TM_TARIFF.dtsx
12. ITC 060 quota 01 into WMDS tar_quota.dtsx
13. ITC 060 quota 02 into WMDO T_TM_QUOTA.dtsx
14. ITC 070 other duty 01 into WMDS tar_ntlc_other_duty.dtsx
15. ITC 070 other duty 02 into WMDO T_TM_TARIFF_ODUTY.dtsx
16. ITC 080 unit value 01 into WMDS tariff_ntlc_uv.dtsx
17. ITC 080 unit value 02 into WMDO T_TM_UNIT_VALUE.dtsx
18. ITC A010 tariffs n-quota and n-specific into D_ADVE_TOT_TARIF.dtsx
19. ITC A020 tariffs quota or specific into D_O_TARIF_new.dtsx
20. ITC A030 tariffs quota or specific into D_I_TARIF.dtsx
21. ITC A040 tariffs join into D_ADVE_TOT_TARIF.dtsx
22. ITC B010 tariffs into FACT_APPLIED_NTLC.dtsx
23. ITC C010 tariffs HS6 APPLIED BY AGREEMENT.dtsx
24. ITC C021 tariffs HS6 MIN MAX.dtsx
25. ITC C030 tariffs HS6 APPLIED MIN MAX JOINED TESTING.dtsx running
ITC C030 tariffs HS6 APPLIED MIN MAX JOINED.DTSX-THIS PACKAGE IS FASTER BUT HAS ISSUES ON
DEPENDECIES WHEN DOING AN ALTER SWITCH. A REVIEW WILL MAKE UPDATES FASTER MEANWHILE
USE PACKAGE NO.25 ABOVE
26. ITC UPDATE HS4 TARIFFS APPLIED.dtsx
27. ITC UPDATE HS2 TARIFFS APPLIED.dtsx
28. WMD E010 TRQ table over all partners Find Tariff.dtsx

If bound tariffs are also updated:

29. WMD D010 bound tariffs with unbound lines.dtsx
If Rules of Origin are provided:
30. ITC RULES OF ORIGIN 01 into T_AGREEMENT_ROO WMDS.dtsx
31. ITC RULES OF ORIGIN 02 into T_AGREEMENT_ROO WMD0.dtsx

1
MACMAP ETL PACKAGES
UPDATED ON 25/09/2012

UPDATING TRADE DATA

The following packages are run in the following order when updating Trade data:

1. ITC 090 01 into WMDS t_trade_hs6.dtsx (if Trade HS6 is provided)

2. ITC 090 01 into WMDS t_trade_ntlc.dtsx (if Trade NTLC is provided)
3. ITC 090 02 into WMDO t_trade FOR HS6.dtsx (this means the source is WMDS.dbo.t_trade_hs6 from steps
above)
4. ITC 090 02 into WMDO t_trade.dtsx (this means the source is WMDS.dbo.t_trade_ntlc from steps above)
5. ITC 090 03 from T_TRADE into WMDC.dbo.FACT_T_TRADE_NTLC( to be ran if NTLC data was provided )
6. ITC 090 trade 04b into WMDC FACT_TRADE_HS6 from WMDO T_TRADE.dtsx (to be ran after ITC 090 02
into WMDO t_trade FOR HS6.dtsx)
7. ITC 090 trade 04c II into WMDC FACT_TRADE_HS6 from WMDO TRADE COMTRADE.dtsx ( to be ran if
COMTRADE DATA WAS PROVIDED FROM THE EXTERNAL TOOL FROM UNCTAD)
8. ITC 090 06 from T_TRADE INTO WMDC.dbo.FACT_T_TRADE_HS4
9. ITC 090 06 from T_TRADE INTO WMDC.dbo.FACT_T_TRADE_HS2

To update the tariff data into the PRODUCTION server (FOR UPDATES IN PROD PLEASE CHANGE THE UPDATE
DATE IN EACH PACKAGE OR VERIFY THAT THE DATA TO BE UPDATED TO PROD IS THE LATEST UPDATE FROM
DEV)
1. ITC 010 agreement type 02 DEV to PROD.dtsx
2. ITC 020 agreement 05 R_WORD DEV to PROD.dtsx
3. ITC 020 agreement 06 R_TRANSLATION DEV to PROD.dtsx
4. ITC 020 agreement 07 T_AGREEMENT DEV to PROD.dtsx
5. ITC 020 agreement 08 T_AGREEMENT_T_AGREEMENT_EUROPE_AGREEMENT_MFN_LATEST DEV to
PROD.dtsx
6. ITC 030 partner agreement 03 T_CONTRACT DEV to PROD.dtsx
7. ITC 040 ntlc description 04 T_NTLC DEV to PROD.dtsx
8. ITC 050 tariffs 04 from T_TM_TARIFF DEV to PROD.dtsx
9. ITC 060 quota 03 from WMDO T_TM_QUOTA DEV to PROD.dtsx
10. ITC 070 other duty 03 from WMDO T_TM_TARIFF_ODUTY DEV to PROD.dtsx
11. ITC 080 unit value 03 from WMDO T_TM_UNIT_VALUE DEV to PROD.dtsx
12. ITC 090 trade 02 from WMDO T_TRADE DEV to PROD.dtsx
13. ITC 090 trade 03b from WMDC FACT_TRADE_NTLC DEV to PROD.dtsx
14. ITC 090 trade 04d from WMDC FACT_TRADE_HS6 DEV TO PROD.dtsx
15. ITC 090 trade 05c from WMDC FACT_TRADE_HS4 DEV to PROD.dtsx
16. ITC 090 trade 06c from WMDC FACT_TRADE_HS2 DEV to PROD.dtsx
17. ITC A010 tariffs n-quota and n-specific from D_ADVE_TOT_TARIFF DEV to PROD.dtsx
18. ITC A020 tariffs quota or specific FROM D_O_TARIFF_NEW DEV to PROD.dtsx
19. ITC A030 tariffs quota or specific from D_I_TARIFF DEV to PROD.dtsx
20. ITC B010 tariffs into FACT_APPLIED_NTLC DEV TO PROD.dtsx
ITC B010 tariffs into FACT_APPLIED_NTLC DEV TO PROD TESTING.dtsx
21. ITC A010 tariffs n-quota and n-specific from D_ADVE_TOT_TARIFF DEV to PROD.dtsx
22. ITC C010 tariffs from HS6 APPLIED BY AGREEMENT DEV to PROD.dtsx
23. ITC C021 tariffs from HS6 MIN MAX DEV to PROD.dtsx
24. ITC C030 tariffs from HS6 APPLIED MIN MAX JOINED DEV to PROD.dtsx
25. WMD E010 TRQ table over all partners Find Tariff DEV TO PROD.dtsx
If bound tariffs are also updated in DEV:
- WMD D010 bound tariffs with unbound lines PROD.dtsx
If Rules of Origin are provided:
26. ITC RULES OF ORIGIN 02 into T_AGREEMENT_ROO WMD0 DEV TO PROD.dtsx
27. ITC UPDATE tariffs HS4 DEV TO PROD.dtsx
28. ITC UPDATE tariffs HS2 DEV TO PROD.dtsx

2
MACMAP ETL PACKAGES
UPDATED ON 25/09/2012

PUBLISHING CONTENT TO DEV AND PRODUCTION

1. PUBLISH CONTENT.dtsx (for dev)
2. PUBLISH CONTENT TO PROD.dtsx

NTM ETL PACKAGES

The NTM LOAD package solution has a text file dependencies.txt UNDER MISCELLANEOUS with further
explanations FOR already existing packages and how to run them.

The load history.xlsx also explains how long each package takes.

The package LOAD NTM data.dtsx will need to be edited to accommodate the new tables created.

New packages for the new module are as follows: new packages(reference C:\Kalif for any
communication and follow up)

1. Migrate NTM data- the purpose was to migrate already existing NTM data to the new table structures
for T_NTM_COLLECTION_NEW and Legislation. No need to re-run it unless you want to correct
already existing data
2. NTM CLASSIFICATION CORRESPONDENCE.dtsx
3. NTM TARIFF CORRESPONDENCE.dtsx- TO be provided by Xavier and be populated before running the
LOAD T_FACT_NTM_HSx.dtsx
4. Load T_FACT_NTM_HSx.dtsx

SHRINKING TEMP DB
DBCC FREEPROCCACHE
GO
DBCC DROPCLEANBUFFERS
go
DBCC FREESYSTEMCACHE ('ALL')
GO
DBCC FREESESSIONCACHE
GO
DBCC SHRINKFILE (TEMPDEV,1024)
GO

3
MACMAP ETL PACKAGES
UPDATED ON 25/09/2012

3.

Packages are designed to automatically rollback actions if any error occurs (TransactionOption = Required on
relevant tasks).

Packages which load txt files into WMDS might need to be changed if the file name changes. This would be
done as follows (below window appears when double clicking on the package):

Deployment
1. Install BIDSHelper a Visual Studio add-in used to set the deployment path in our scenario.

4
MACMAP ETL PACKAGES
UPDATED ON 25/09/2012

2. Create a directory junction on SCOTS between e.g. C:\WMD_DATA\03_Scripts on your workstation

and the same folder on SCOTS.

3. Right click on the project and click deploy: it will copy packages into the specified folder on your
workstation.

4. Then, thanks to
the directory junction

5
MACMAP ETL PACKAGES
UPDATED ON 25/09/2012

these packages will also be visible on SCOTS.